The goal here is to design an IC that has a simple SPI-slave interface to the rest of the world, and a SHA-3 encryption engine (Keccak) internal to the chip. You'll be able to feed it bytes of data into a (small) FIFO over SPI, tell it to 'go', and have the chip produce a cryptographic hash of the input data. There's nothing world-shattering here - let's leave those expectations at the threshold. Adages about not trying to run before you can walk come to mind, and I think this is plenty adventurous for a first trip into the unknown. I'll be targeting a 180nm technology, for which there are publicly available standard cell definitions.
The steps necessary will be:
Investigate and make ready for use the software that we're going to need in order to actually go from an FPGA HDL (verilog by choice) to an ASIC mask description.
Investigate and ensure that it's even possible for an individual to submit to MOSIS or a similar MPW "shuttle" service vendor. I don't mind setting up a company to do it, but it'd be less hassle to do it just as me.
Write the HDL source code, simulate it, and verify it works on an FPGA
Learn the tools and technologies that we'll be using in order to produce the ASIC mask
Actually use the tools and make the mask
That last one seems rather a short definition of a rather large problem…