Daily Reading Notes

January 22, 2021

The Art of Doing Science and Engineering

Learning to Learn

Style separates the great from the good.

You cannot learn style from a book or a class. You have to study the style of other's and apply your own. Then you will start to develop a style unique to you.

What's successful in one age will not succeed in the next.

"Education is what, when and why to do things.
Training is how to do it."

In order to understand some things, you have to experience them.

History repeats itself outside of radical disturbances, like computers.

Math and technology hold the potential to radically change our future.

Society, culture, and politics will ultimately shape our future.

Back of the envelope calculations can quickly determine if you're on the right track.

Learn to model verbal arguments using mathematics or programming. These models will give you the basis to manipulate arguments to come up with new insights.

Knowledge doubles every 17 years. This means you can never stop learning.

To learn better, focus on the fundamentals.

Fundamentals either lasted a long time, or act as the foundation for the rest of the field.

"In science, if you know what you are doing, you should not be doing it.
In engineering, if you do not know what you are doing, you should not be doing it."

Engineering and science continue to move closer to each other.

Without a vision you lose all effectiveness.

Even a simple vision will serve you better than no vision.

Consider three questions when planning:

"What is possible?
What is likely to happen?
What is desirable to have happen?"

Computers have advantages over people just as people have advantages over computers. Learn when to leverage the correct tool.

Discrete signals allow getting to the key information by eliminating noise. This makes it far easier to transmit information accurately and continuously.

Society continues to move towards valuing information more than material goods. You can see this by the numbers of people involved in producing material goods versus the number of people involved in working with information.

Computers allow for the production of better, cheaper, and different goods.

As things become automated, the details of production will change. This will require a flexibility in the detail while keeping the big picture in mind.

Technology innovation growth as an S curve. Development and innovation starts off slowly. Then comes a period of nearly exponential growth. With the life cycle ending in a stagnation before the next innovation.

Use general purpose solution wherever possible. They benefit from the ideas and testing of many more users than just yourself.

You may require a specialty solution sometimes. Try to make that time as short lived as possible.

We have just scratched the surface of the effect computers will have on our society.

Modeling problem using computers allows running more experiments faster, and even running experiments that would not be possible otherwise. To fully prove an outcome, however, you will need to run a real world experiment.

Central management and low level micro management have too wide and too narrow of a view respectfully. This leads to poor decision making and underwhelming outcomes.

Small, mostly independent, organizations are starting to loosely connect with each other in order to combat the inefficiencies of single large organizations.

Huge value lies in the experiences and thoughts of the people doing the job.

You cannot blindly trust anyone or anything. Be selective about what you choose to believe.

Even the speed of light only goes so fast. Because of this limitation, computers hit a limit to haw fast they can process data.

We are approaching the limit of the speed achievable by a single processor. To continue increasing compute, we must explore highly parallel architectures.

There is no foreseeable standard for parallel architecture, therefore we will make many wrong turns before reaching a the growth portion of the S curve.

People will always resist change, and claim that "real x" don't do it that way. If we always followed that logic we will never make major progress.

Programming is ultimately about symbol manipulation.

We continue to virtualize more of the machine. This allows for portable programs that can work anywhere.

The inventors of a new idea rarely understand it's full implications. It takes other people using their idea to realize it's full potential.

We should strive to make it possible for the domain experts to write as much of the program as possible.

We can't make human computer interaction better until we gain more understanding of human to human interaction.

Languages designed with the psychological side in mind will surpass languages that put logic first. Human have to use the languages to write programs, and ultimately, all humans are psychological creatures.

To create a good program, you must think of a plan before proceeding but remain flexible to finding a solution as you go.

Writing software is closer to writing a novel than engineering. It takes a lot of creativity and evolves throughout the process.

Just like it's hard to train a good writer, it's hard to train a good programmer. Pay the good ones well and fire the bad ones.

A good scientist must perform three duties:
1. Write papers and books
2. Give public talks
3. Provide impromptu talks

You can only gain these skills by practice.

An innovation must prove that it can do things that were not possible before. If that's too high of a task, then it is highly unlikely that it will gain adoption in the long run.

After an innovation gains adoption, you can start using it to improve existing processes. Often times, more value comes from this than the original innovation.

You will only see a steady growth rate in a field if new innovations continue to appear before the previous innovation starts flattening out.

The next field of major innovation comes in pattern recognition. Computer can help us quickly identify everything from written language and images, to sounds and signals.

Providing a progressive output can greatly increase decision making capabilities.

We rarely need live data. Often time better decision making is enabled when everyone has the same data properly aggregated.

Software provide the ability to create specialty systems on commodity hardware.

Expert systems only work in some instances. We still don't fully understand the limitation of these or any other rule based systems.

You have to ask "to what extant can computers think?" Then see if you can push that further.

Everyone has a different definition of "machine" and "thinking". We don't know our real progress in the field of AI because it's still an ill defined field.

The Turing test is not scientific because it only states the final state and no progression towards it.

When we engineer things they do things differently than even the creators would expect.

Because algorithms execute perfectly as directed, they can come up with solutions that a human would not think of.

These novelties are only psychological. The computer merely executes it's instructions perfectly. As humans, we tend to take shortcuts.

If we pit a program against itself while only modifying decision making parameters and progressively improve it by picking the winning combinations, did the program learn?

The biggest advantage AI can give us comes in working with people, not against them. Just look at tools that help with photo editing and music productions. People could never achieve that amount of precision without the help of a corrective AI.

Computers can help in complex decision making, like a medical diagnosis, but where would we place the blame for a mistake? With a doctor, we can accept their mistake because they are only human. Can we accept the same in a machine? After all, it was only designed by humans.

It's ridiculous to depend on randomness to eventually come up with the correct answer. The process would take forever.

Do computers have free will? For that matter, to we have free will? Our environment, other people, and marketing has a bigger effect on us than we care to admit. Is this so different from the effect we have on computers?

We have poor intuition for the reality of multi dimensional spaces.

Every parameter that we add to a problem adds a new dimentions to our problem space. This means that any non trivial program is a multi dimensional space problem.

Every useful program starts with a source and end with a sink.

A program's first step will involve decoding the information it receives from the source.

Then the program must deal with it's input and any system or environment noise.

After transformatin it's input, the program must encode in the format expected by the sink.

This process stays the same no matter if the result is sent to anothr process or stored in memory. Only the encoding and the sink will change.

It is impossible to build a system that is always correct, especially as systems grow more complicated. Instead we must build error correction into an imperfect system.

The more complex the system becomes the higher the maintenance costs gets. Error correction becomes the best way to keep the maintenance costs manageable.

You will get many opportunities for greatness, but you will only succeed when you are sufficiently prepared.

Sometimes you may not know what you're preparing for. None the less, it's your duty to prepare for when opportunity comes.

Your expected outcome can greatly influence what you find. You may overlook important details, or influence the result through the tools you choose to use. This is why it's important to have others verify your findings.

When learning about a new field or tackling a new problem, start with the very basics. This foundation will give you a solid start to build onto.

You should always do the important work, even if you don't have the desire to do it. These types of efforts become exceedingly rewarding in the long run.

Anything that improves things by an order of magnitude has a major impact on the future. Get involved with these types of opportunities if you want to make meaningful contributions.

As the amount of information grows teamwork becomes increasingly important. You must learn to work well with others.

Do not chase credit for the work you help accomplish. Remain humble and allow others to publish and take the credit. In time, they will give you credit and that will go much further than claiming it for yourself.

Use quick simulations to quickly test guesses for improvements to an imperfect solution.

Remember why you determine something to be impossible. The next time the problem comes around, carefully review your reasoning. Technology might have changed enough to make it possible.

You can learn a lot from black boxes by analyzing only the inputs and outputs.

To get good results you must adjust for delays. Either decrease your response to sudden changes, or delay your sampling.

To produce breakthrough results you must strike a balance between questioning basic facts and looking forward to the next advance. Insight can arise in either areas.

As experts become more specialized, their view narrows. Therefore, multiple experts working together need someone around that can keep the big picture in mind course correct as needed.

Even the best expert in simulation needs to work with a domain expert. Without the knowledge of the field, they can miss critical points of the problem.

Always start with the simplest possible simulation. This will give you fundamental insights into the working of the problem. Once you gain the insight, move towards making the simulation more realistic.

Every field develops a jargon. It helps with communication in the field, but excludes outsiders.

Jargon is a symptom of our early evolution to distrust outsiders. If we want to work well in large teams, however, we must move past it.

How have you tested your simulations? Why should anyone trust them?

Beware of trying to simulate situation where it's possible to cheat -- like the stock market, games, or gambling.

The older the audiance, the harder time you will have communicating a new idea. Minds become rigid with age.

Certain parameters might not need accuracy or concern. Garbage in can produce accurate enough results if the data converges.

Sometime the final result isn't the most important piece of the simulation.

Medical studies need to run double blind studies to prevent the doctors and patients from favoring the outcome they believe in. Are simulations any better off?

Before you claim to see a pattern, rule out randomness.

To make good prediction about a new scientific advancement, follow this process:

- Learn everything you can about the current developments.
- Identify any challenge areas.
- Try to predict where the technology could go in the short term and long term.
- Run your prediction through a social, economic, and political filter.

Learning has no shortcuts.

Pattern recognition will benefit you far more than memorization.

Advancements in a field rarely come from the experts. They come from people that can combine knowledge from multiple fields into something entirely different.

The Hawthorn effect shows that the novelty of a new method can boost learning in the short term. Long term, as the novelty wears off, the method could become worse.

Math is only useful becomes it maps to real world ideas.

Theorems are often derived backwards. The mathematicians or scientists start off knowing the truth and work their way into the equations.

Math creates a circular problem. The symbols and formula manipulation are useless without underlying physical representation, but the physical phenomenon cannot be fully understood without the use of formula and symbols.

As our understanding grows and we learn to explain parts of the real world, we discover exponentially more parts that need explaining.

The easy to explain things have already been explained. As we progress, we will need to have fuzzier explanations for more complex things.

Any program that's useful is too complex to prove, just like most of the useful theorems. Instead, programs help define the domain better and vice versa.

Quantum Mechanics presents us with a field that we have failed to understand for over 70 years. Despite us lacking complete grasp, however, we managed to come up with mathematical models that help us reason about the phenomena in the field.

As we continue to progress to discover more complex fields, we will have more concepts whcih we can only use through mathematical models. This fact does not make the fields any less useful than well understood ones.

Novelty and originality do not equate to creativity. Creativity requires those things, but goes on to add usefulness.

Creativity involves putting together things that were previously thought unrelated.

Coming up with a solution requires a long period of thinking about the problem. Followed by a likely break from the problem. Until finally the subconscious comes up with a solution.

The failure of one solution narrows the number of possible solutions and makes a successful one more likely to come next.

Coming up with the solution rarely means that your work ends. You will have to refine and clean up the solution until it's ready for presentation or implementation.

In order to foster creativity you must have a breadth of knowledge to draw from. In order to build up this knowledge, think deeply on new ideas. Come up with analogies, think of new uses for the concept, and apply the idea to things you already know.

Any change of habit is hard. Start slow and make gradual but consistent progress.

If you can't learn to drop the wrong problems, you will never have time for the right ones.

Experts usually say that a new approach is impossible because it would require them to change their underlying assumptions. It's extremely hard for anyone, especially experts, to admit that they have been wrong all along.

Always test your fundamental assumptions and make a conscious decision to reject or try a new idea. Do not allow your assumptions to just drag you along.

When you think you know something absolutely, ask yourself: "What would it take to change my mind?"

The same approach will not work forever. Eventually it will become counter productive.

If you cannot update your mental models with the progress of the field, you will lose the joy in your work, and get left behind.

Copyright © Artem Chernyak 2020