Google is worried as the world goes mobile. As you may be reading now—on a smartphone—the future of your mobile systems is threatened by how tech giants and academia together lack deep research into the computer architecture of mobile devices.

Because of this inadequate study, Google has written a “ten commandments” for what industry and universities should do when researching mobile workloads, metrics, and experimental methodology.

At stake is the future innovation and growth of mobile devices, now numbering 2 billion worldwide, compared with a global population of more than 7 billion.

“We believe that the dearth of research on mobile computer architecture fundamentally stems from the severe lack of knowledge about how mobile devices are used on a daily basis, how they perform ‘in the wild,’ and what challenges the industry is facing with mobile computer architecture design,” the Google team writes for the IEEE Computer Society.

“Why is it that only 1 percent of all architecture research papers published each year in the top computer architecture conferences focus on mobile computing? Surely it can’t be that the architecture community thinks that mobile computing is less important than datacenter computing,” write the researchers in “Two Billion Devices and Counting” (login may be required for full text) in the society’s IEEE Micro magazine.

“Taking all of the major cloud service providers—Google, Amazon, Microsoft, and Facebook—into account, a liberal estimate of the number of servers worldwide is perhaps 10 million. So, the current smartphone-to-server ratio is 200:1,” they write.

The mobile market is a bonanza. Though North America and Europe are reaching saturation, other parts of the world are exploding, especially Asia. Mobile subscriptions are projected to reach 6 billion worldwide in coming years, the Google team says.

The authors of this research—and ten commandments—are Vijay Janapa Reddi, a visiting research scientist at Google working in the Mobile SoC Architecture team who has been on leave from his associate professor position at the University of Texas at Austin; Hongil Yoon, an SoC performance architect at Google; and Allan Knies, who is the simulation and performance lead in Google’s hardware division.

They contend that if current challenges are addressed adequately, the entire mobile ecosystem will advance to the next level. They wrote a ten commandments for experts to ensure a path there.

Read prior Computer Society story about how Google offered “152 Simple Steps to Stay Safe Online.”

Commandment #1: Thou shalt not rely solely on benchmarks.

Testing the features of computer programs is an essential part of development. However, the standard industry benchmarks used to do this are proving inadequate in mobile computing.

“In contrast to many of the steadfast benchmarks, which are composed of a handful of select applications, the mobile application ecosystem is diverse and the application’s characteristics change overwhelmingly fast. After ten years of mobile computing, there are more than 3 million applications currently in the Google Play Store. The number of applications available first exceeded 1.5 million in 2015. This means more than 60,000 new applications are released in the Google Play Store each month. Keeping pace with the mobile ecosystem is challenging because benchmarks evolve more slowly than real-world applications,” the authors say.

Although it poses a whole new set of challenges, benchmarks need to keep pace with the applications using them.

“A strong call to action is the development of a benchmark suite that faithfully represents real-world application characteristics and user interactivity,” say the authors.

Commandment #2: Thou shalt not cherry-pick applications.

Choosing applications to test benchmarks can be hit or miss. Basing selection on popularity isn’t enough.

“It is typical practice in the community to select the top applications from the app stores to conduct detailed micro-architectural analysis studies. But architects must learn to be cognizant of the aforementioned pace of change, as the popularity of mobile applications can evolve quickly,” the authors say.

The graph below shows the popularity trend of three different apps: Google Chrome, Angry Birds, and Pokémon GO. As you can see, the popularity of an app doesn’t necessarily mean it will last.

Commandment #3: Thou shalt not ignore the web browser.

Virtually all smart phones now come with a web browser. However, many people believe that the quality of the network determines how well the browser works. This isn’t true.

“Many assume that user experience while browsing is solely affected by network performance. However, advancements in LTE network and WiFi speeds have caused the browser to be more compute-bound. As such, the browser relies heavily on the CPU for its rendering and network processing,” the authors say.

The browser is an integral part of any benchmark development.

“Mobile processor architects must include the browser in the creation of a benchmark suite and any sort of hardware or software optimization analysis,” say the authors.

Commandment #4: Thou shalt not drop a frame.

We use our smart phones all the time, but we might not realize how much.

“An average user taps, types, swipes, or clicks her or his device 2,617 times a day, and about 10 percent of us perform those actions 5,427 times a day. So, what matters to users is the touch responsiveness of the system (the time it takes to render a frame after a touch input). To ensure ‘buttery smooth’ responsiveness, the system architecture as a whole must maintain 60 frames per second (FPS) consistently without any dropped (or delayed) frames, commonly referred to as ‘jank,’” say the authors.

To improve smart phone testing, the authors recommend reporting dropped frames in simulations.

“Given the criticality of responsiveness for user experience, it would be prudent for mobile architectural simulators to report the frames that have been successfully processed and those that have been dropped per second, in addition to the more traditional raw runtime and microarchitectural efficiency metrics of the processor,” the authors say.

Commandment #5: Thou shalt not idolize micro-architectural efficiency.

The authors recommend not wasting time on improvements that don’t also improve user experience.

“Doubling the translation lookaside buffer (TLB) from 32 to 64 entries or increasing CPU clock frequency might seem like the right trade-off to improve performance at the expense of power consumption. In practice, however, if the micro-architectural enhancements cannot translate to measurable user experience, doing so won’t be useful,” they say.

In the chart below, you can see how improving energy efficiency improves user experience only up to a certain point:

“By dropping from 2.15 GHz to 1.67 GHz, we can still meet the 16.67-ms cut-off for all the frames, save a significant amount of power and energy, and run cooler. Put another way, increasing the frequency from 1.06 GHz to 1.67 GHz is worth the extra energy, as it helps meet the cut-off latency. Pushing beyond 1.67 GHz can be wasteful in terms of performance and energy,” the authors say.

Commandment #6: Thou shalt not ignore tails in user experience.

Even if it takes a short while for an app to load, we can tolerate it if it happens consistently. We find it intolerable, though, if load time varies wildly—taking a moment or ten times as long. “Tail latency” illustrates this frustrating unpredictability.

The chart below shows a Google onCreate user event from two first-party Google applications, both of which have at least a few hundred million users.

Because data is pulled from numerous storage devices in response to a request, it’s hard to solve the problem of tail latency. However, according to the authors, the information should at least be reported.

“There is no well-known way to control this variability, because there are too many sources. This means that to have confidence in whether any architectural optimization has impact, the optimization must first withstand the variance in the execution times. Otherwise, we do not know if the improvement is truly from the experiment or if it’s by chance. So, often, the safest way is to report the distribution or assume the worst case and report results accordingly,” say the authors.

Commandment #7: Thou shalt not assume software is hardware-agnostic.

With more than 24,000 different Android devices on the market, it’s astonishing how an app like the Chrome browser can work seamlessly in so many systems. But it does. The next generation of smart phones will demand even greater performance. That’s why tech companies must hire people who know what they are doing.

“Because hardware-aware software tuning isn’t rare in the mobile ecosystem, when we perform comparative performance analysis between any two platforms, we need to be aware of such crucial differences. Sadly, however, this requires an inscrutable level of domain-specific application level knowledge. But that is the price to pay for working in a rich and highly heterogeneous system,” the authors say.

Commandment #8: Thou shalt not disregard the IP blocks.

Often, computer architects focus too much on development of the CPU to the exclusion of the IP blocks (which operate concurrently to handle tasks), or they study IP blocks in isolation instead of as part of a system. The authors say this is a mistake.

“The magnitude of concurrency in a mobile system should reinforce the need for computer architects to understand the system as a whole, and not focus solely on the CPU subsystem. The brunt of the work in a mobile device is handled by the IP blocks, and the CPU generally plays a minimal role, mostly coordinating activity between the different IPs. Therefore, architects are strongly encouraged to consider the role of the various IP blocks in the process of evaluating a mobile system. There are numerous opportunities for research and development of novel solutions to enhance mobile processor performance,” they write.

Commandment #9: Thou shalt not turn a blind eye to energy and thermal.

Some things we do with our mobile devices require far more power than others. For example, recording video can take up to five times more power than simple web browsing. As mobile devices are being equipped for even more advanced tasks like machine learning and augmented reality, the need for more power via longer-lasting batteries and heat dissipation grows too.

“The differences in the power consumption and activation of different IP blocks indicates that we need to optimize the system as a whole by considering power, thermal, and performance together. We expect this trend will continue as more IP blocks and custom accelerators are integrated into the SoC for emerging use cases, such as machine learning and augmented reality. Therefore, measuring and quantifying the power peaks and energy consumption of new ideas is necessary given that battery and heat dissipation are first-order constraints for mobile systems,” the authors say.

Commandment #10: Thou shalt not presume this list is complete.

Okay, so there aren’t really just ten commandments, but the authors say it’s a good place to start.

“Mobile computing is a rapidly and continuously evolving ecosystem, and the hardware requirements will need to evolve as time progresses. It would be presumptuous for anyone to assume that any one set of commandments can ever be complete. To this end, the tenth commandment acknowledges that the current list is by nature incomplete and that the aforementioned commandments should be improved upon over time as mobile computing evolves,” they say.

Related research on smart phone tech in the Computer Society Digital Library:

(login may be required for full text)

• On-Device Mobile Phone Security Exploits Machine Learning
• Insecure by Design: Protocols for Encrypted Phone Calls
• The Mobile Phone as Surveillance Device: Progress, Perils, and Protective Measures
• You Are Your Cell Phone
• FTC and Apple Sue Qualcomm for Cell Phone Standardization Skullduggery: Part 1
• FTC and Apple Sue Qualcomm for Cell Phone Standardization Skullduggery, Part 2: Apple’s Claims
• FTC and Apple Sue Qualcomm for Cell Phone Standardization Skullduggery, Part 3: Determining SEP Reasonable Royalty
• FTC and Apple Sue Qualcomm for Cell Phone Standardization Skullduggery, Part 4

About Lori Cameron

Lori Cameron is a Senior Writer for the IEEE Computer Society and currently writes regular features for Computer magazine, Computing Edge, and the Computing Now and Magazine Roundup websites. Contact her at l.cameron@computer.org. Follow her on LinkedIn.