Pay attention to the “shouldn’t” in the title. Of course you can break these rules, and you might find very good reasons to do so. But if you’re new to visual design, here are some visual design rules you can safely follow all of the time.
Pure black looks unnatural on a screen, and pure white is too bright. Use a vey dark and very light grey, respectively. Any other references to “black” and “white” in these rules assume you’re following this rule.
A neutral is generally a black, white, or grey. If you use colour in your interface, add a little bit of that colour to your neutrals. If you use the HSB colour system (and you should) less than 5 Saturation should do it.
Structural elements are those that make up the layout of the page. If you have a card, for example, that’s a structural element. So is the page background. For these structural elements you should use colours that contrast a little with each other. In practice this means if your background is black the card should be slightly less black, not white. The content, on the other hand, should contrast well with the structural elements. If your card is black, the text inside it should be white.
This is vague, but very important. You should be deliberate about absolutely everything in your design. This means whitespace, alignment, size, spacing, colour, shadows. Everything. If I point at a random part of your design and you don't have an explanation for why it looks that way, you’re not finished.
Mathematical alignment is when you tell your design software to centre something in its container and think you’re done. But some shapes don’t suit being aligned in this way. Very often you will need to align things by eye so that it looks good.
This applies to all text. The bigger the text, the less space you need between each letter and each line.
The default settings for drop shadows in design software are often too hard/dark. Make your drop shadows as subtle as you can.
Usable elements are anything that a user is supposed to notice, take in, or interact with. You will be fine if you follow the colour contrast guidelines set out in the WCAG. If an element is there purely for decoration this rule does not apply.
Example: If you have a card with a 1px border and a dark background, and it sits on top of an even darker background, the 1px border should be lighter than both of them. It should not be set to a brightness somewhere between the card and page background colours. The same applies to light background colours: the 1px border should be darker than both background colours.
If something is not aligned with anything else in the interface, it looks terrible. Ideally each element will be aligned with other elements based on some kind of logic.
It’s OK if some colours in your palette share the same brightness value, but if a colour has a different brightness value it should be noticeably different.
Having some blacks with a little bit of warm colour and some with cool colours looks bad.
The spacing you use between elements, and the size of elements, should be determined by some kind of scale. This will help the design to look consistent. An easy way to do this is a column grid and a baseline grid.