Cache Memory – Everything You Need To Know

As the performance gap between processors and main memory continues to widen, increasingly aggressive implementations of cache memories are needed to bridge the gap. Cache memory is a small fast memory used to temporarily hold the contents of portions of the main memory that are most likely to be used.

In computing terms, Cache refers to the software or hardware component that holds data to serve the future demands of that particular saved data a lot faster. The stored data can also be that might have been copied or saved elsewhere for protection purposes. When the requested data is found in a cache, a cache hit is said to occur. Meanwhile, when the requested data is not found in a cache, a cache miss is said to occur.

Cache hits are super-fast and occur quicker than recomputing as a result of reading through a much slower data store. In simple words, the more the data search requests are implicated on cache, the faster it gets to serve and provide super-fast results.

Today, caches and cache memory have become an vital component of all processors.

To make caches more efficient and effective to store data and serve data search requests, caches must be relatively smaller in size. By making them smaller in size, it provides one more benefit to the user and, i.e., cost-effectiveness. Cost-effectiveness is a benefit which every user looks for in every aspect.

The ability of caches to bridge the performance gap is determined by two primary factors, the time needed to retrieve data from the cache and the fraction of memory references that are and can be satisfied by the memory cache. These two primary factors are commonly referred to as ‘‘access (hit) time’’  and ‘‘hit ratio’’.  

The computer has three logical systems, the CPU, the memory and storage system, and the input/ output system. Cache memory has significantly advanced in recent years ever since Wilkes proposed a ‘‘two-level main store’’  in 1965. The first being Conventional slave memory and the second being Unconventional slave memory.

It has now become a conventional component of high-speed computing with increasing size and sophistication. The performance of cache is critical to overall system processing ability and the ongoing research in this area is attempting to reduce the speed gap between the CPU and the cache memory as much as possible. 

The Different Types of Cache memory aspects –

1. Cache Fetch Algorithm- The cache fetch algorithm is used to decide when to bring the information to the cache. Several possibilities exist where information can be fetched on-demand when it is needed or before it is needed that is prefetched. 
2. Cache Placement Algorithm- Information is generally retrieved from the cache associatively because large associative memories are relatively slow and quite expensive. Hence, the cache is organized in relatively smaller associative memories. Thus, only one of the associative memories has to be searched to determine whether the desired information is available in the cache or not. Each of this small associative memory is called a ‘‘set’’  and the number of elements over which the search is conducted is called set size. 
3. Line Size- The fixed-size unit of information transfer between the cache and main memory is called the line. The line corresponds to the page, which is the unit transfer between the main memory and the secondary storage.
4. Replacement Algorithm- When information is requested by the CPU from the main memory and the cache is full, some information in the cache must be stored for replacement.
5. Main Memory Update Algorithm- When the CPU stores a memory, that operation can be reflected through copy-back and write-through in multiple ways. 
6. Supervisor Cache- The frequent switch between the user and supervisor in most systems results in high miss ratios because the cache is often reloaded.
7. Input/ Output- Input and output are additional sources of reference to the information of memory. It is important that an output request stream reference the most current values for the information transferred. Similarly, the input data must be immediately reflected in all copies of those lines in memory.
8. Data/ Instruction Cache- Another cache strategy is to split the cache into two parts- one for data and one for instructions.
9. Virtual vs Real Addressing- In systems with virtual memory, the cache may be accessed with real addresses or virtual addresses.
10. Cold start vs warm start- Most systems are multi-programmed, that is, the CPU runs several processors. While only one can run at a time, as they alternate every few nanoseconds. 
11. Multi-level Cache- As the cache grows in size, there comes a point where it splits into parts- a small, high-level cache that is faster and smaller and a larger second-level cache.
12. Cache Bandwidth- It is the rate at which the data can be read from or written to the cache. 

THE FUTURE PROSPECTS OF CACHE AND CACHE MEMORY

New memory technologies are blurring out the previously performed characteristics of adjacent layers in the memory hierarchy. There are no longer such layers of orders of magnitude which are different in request capacity or latency. Beyond the traditional single-layer view of caching, there exists a data placement challenge. An offline algorithm for data placement across multiple tiers of memory with asymmetric read and write costs exists called CHOPT.

It is optimal and can therefore serve as the upper bound of performance gain for any data placement algorithm. The ACM demonstrates an approximation of CHOPT which makes its execution time for long traces practical using special sampling of requests incurring a small 0.2% of an average error on representative workloads at a sampling ratio of 1%. 

An important to note is that in the near future, static energy will dominate the energy consumption in deep-micro processes. In the stimulation using SPEC95 integer benchmarks, their technique used about 45% of leakage memory in the cache at maximum, and about 28% on average. Their results identify substantial improvement opportunities for future online memory management research for developing further memory cache aspects in the future.