Humans are intelligent animals, but humans are not only intelligent animals. We do not make decisions and choose which beliefs to hold based solely on sober analysis of facts.
That doesn’t change the general point that a model given the vast corpus of human knowledge will prefer the most oft-repeated bits to the true bits, whereas we humans have muddled our way through to some modicum of understanding of the world around us by not doing that.
But the most current information does not mean it is the most correct information.
I could publish 100 papers on Arxiv claiming the Earth is, in fact, a cube - but that doesn’t make it true even though it is more recent than the sphere claims.
Some mechanism must decide what is true and send that information to train the model - that act of deciding is where the actual intelligence in this process lives. Today that decision is made by humans, they curate the datasets used to train the model.
There’s no intelligence in these current models.
The raw spending figure isn’t what is important, but the PPP figure. Russia’s economy is about 1/5th the size of the EU’s in PPP, and its defense sector is vastly more efficient on a monetary basis than the west - The US alone has given Ukraine close to $60 billion and it is a fraction of the hardware that Russia has produced with fewer dollars.
This isn’t a ‘Russia stronk, Europe bad’ post, it just bears emphasizing that Russia has a large industrial base and has brought much of it into arms production over the past two years. The West hasn’t, and defense procurement remains an almost artisanal process where high tech goods are bought - in low volumes - at inflated prices.
The terms seem agreeable?
The terms that restrict the size of the Ukrainian military, bar Ukraine from receiving foreign assistance to rebuild its military, forbid it from seeking security guarantees from any country or bloc, … The terms that would have made it trivial for Russia to further invade at any point in the future?
Those terms seem agreeable?
A token is not a concept. A token is a word or word fragment that occured often in free text and was assigned a number. Common words, prefixes, and suffixes are the vast majority of tokens, and the rest are uncommon pairs of letters.
The algorithm to generate tokens is essentially compression, there is no semantic meaning embedded in them.
Copilot is GPT under the hood, it just starts with a search step that finds (hopefully) relevant content and then passes that to GPT for summarization.
Every billion parameters needs about 2 GB of VRAM - if using bfloat16 representation. 16 bits per parameter, 8 bits per byte -> 2 bytes per parameter.
1 billion parameters ~ 2 Billion bytes ~ 2 GB.
From the name, this model has 72 Billion parameters, so ~144 GB of VRAM
Role of thumb is an employee costs roughly twice their base salary, as the employee still needs to cover insurance, taxes, sick time, and other benefits.
That leaves an average salary of 190K for the 50 employees. That isn’t much for tech.
Explaining what happens in a neural net is trivial. All they do is approximate (generally) nonlinear functions with a long series of multiplications and some rectification operations.
That isn’t the hard part, you can track all of the math at each step.
The hard part is stating a simple explanation for the semantic meaning of each operation.
When a human solves a problem, we like to think that it occurs in discrete steps with simple goals: “First I will draw a diagram and put in the known information, then I will write the governing equations, then simplify them for the physics of the problem”, and so on.
Neural nets don’t appear to solve problems that way, each atomic operation does not have that semantic meaning. That is the root of all the reporting about how they are such ‘black boxes’ and researchers ‘don’t understand’ how they work.
Haskell isn’t really that hard to learn
A monad is a monoid in the category of endofunctors
In many cases, getting something out quickly is more valuable than having it be clean.
Part of being a senior is knowing when fast is more important than perfect. Not saying your senior did everything right, just that a single example of someone’s code isn’t enough to judge the value of a person to an organization.
Most OLEDs today ship with logo detection and will dampen the brightness on static elements automatically.
While it isn’t a silver bullet, it does help reduce burn in since it is strongly linked to heat, and therefore to the pixel brightness. New blue PHOLEDs are expected to also cut burn in risk. Remember that LCDs also used to have burn in issues, as did CRTs.