Geneious Prime runs in a Java Virtual Machine. When this JVM starts, it will be allocated a certain amount of RAM and the program can use less than that but never more. In Geneious R8 and higher on Windows and Linux machines, and Geneious 10.1 and higher on macOS, an appropriate amount of RAM based on your total RAM will automatically be allocated. In Geneious R7 and earlier, and on mac versions prior to 10.1, the default memory allocation is 1GB of RAM on 64-bit machines, and 700MB of RAM on 32-bit machines. To change the amount of memory allocated, go to Tools→Preferences, and in the General tab increase the Max memory available to Geneious Prime setting. On Windows you need to run Geneious Prime as an administrator to change this setting (to do this, right-click on the Geneious icon and choose “Run as administrator”).
On a 32-bit machine you can only allocate a maximum of 1GB, so if you need more RAM than this you’ll need a 64-bit machine. You should never allocate the total memory of your computer to Geneious, as you need to leave some RAM available for your operating system. As a general rule of thumb, on a 64-bit machine with up to 8GB of RAM in total you can allocate half the RAM to Geneious. On machines with more than 8GB RAM you should leave 3-4GB spare for your operating system and allocate the rest to Geneious.
To see how much memory Geneious Prime is using, check the Memory Usage Bar under the Sources panel. You may need to turn this on under Tools → Preferences → Appearance and Behavior. Clicking this bar will force a garbage collection and free up memory within the JVM.