Google putting its trust in Rust to weed out memory bugs in Android development
- Reference: 1617799091
- News link: https://www.theregister.co.uk/2021/04/07/google_rust_android/
- Source link:
The Android project has largely been built in two languages. Java (and more recently, JVM-compatible languages like Kotlin) have been favoured for higher-level parts of the operating system, such as the UI. OS fundamentals, like the kernel and drivers, have typically been written in C, and, to a lesser extent, C++.
[2]
C and even C++ are considered well suited for system-level programming as they offer a degree of closeness to the underlying hardware that's hard to achieve with higher-level languages. There are no intermediate layers of abstraction, like the Java virtual machine. C is also highly portable, and developers are left to their own devices when it comes to things like memory handling.
But this flexibility has repeatedly proven to be a double-edged sword. With no garbage collection to rely on, simple memory management errors frequently result in serious security issues, such as buffer overflows and overreads. According to Google, memory-safety bugs represent 70 per cent of all high-severity security vulnerabilities found in the Android Open Source Project.
[3]
Introduced in 2010 by Mozilla, the Rust programming language provides a happy medium between the low-level power of C and the memory safety found in higher-level languages like Java.
But its approach is novel. Rust doesn't place the same emphasis on manual memory management like C. Nor does it come with a Java-style garbage collector (although the existence of a Rust garbage collector is a [4]somewhat contentious topic ). Rather, it takes a third tack, with memory usage linked inextricably to scope.
If Google and Apple won't help us, we'll sort it out the Linux way: 21 companies form Mobile Native Foundation [5]READ MORE
A core concept of Rust is ownership. Put simply, every value has an owner. When that owner goes out of scope, the value is dropped. This limits the risk of memory-based security bugs, while also limiting the amount of code needed to be written to ensure safety.
Additionally, Rust requires all variables be initialised before use, preventing an issue that Google claims is the root cause of between 3 and 5 per cent of all Android security vulnerabilities. Rust also comes with native protections against buffer overflows, another memory security woe.
Google has said it doesn't intend to start rewriting the entire OS in Rust, which would be a mammoth undertaking given the millions of lines of code within Android. Rather, it intends to use Rust going forward. This, it said, is due to the disproportionate amount of memory-safety errors in newly written code. Vintage code is comparatively more robust, with errors already weeded out.
Android's adoption of Rust won't happen overnight. Describing the task as "a large undertaking," Google highlighted the vast amounts of underlying work needed to be done. "There are toolchains and dependencies that need to be maintained, test infrastructure and tooling that must be updated, and developers that need to be trained," it said.
Scaling the language to widespread use will be a "multi-year project," although the company has some early-adopter projects in the pipeline that rely on Rust.
[6]
Android's embrace of Rust is indicative of the language's growing popularity. In July last year, Linus Torvalds suggested Rust [7]may find a place in the Linux project , particularly with respect to device drivers. This preceded the porting of the [8]GNU Coreutils suite (which is a suite of basic file, shell and text manipulation utilities) to Rust in 2020 by Debian developer and Mozilla director Sylvestre Ledru. ®
Get our [9]Tech Resources
[1] https://security.googleblog.com/2021/04/rust-in-android-platform.html
[2] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/personaltech&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=2&c=2YG3XJ2YqybwG27iTowKwlQAAABM&t=ct%3Dns%26unitnum%3D2%26raptor%3Dcondor%26pos%3Dtop%26test%3D0
[3] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/personaltech&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=3&c=33YG3XJ2YqybwG27iTowKwlQAAABM&t=ct%3Dns%26unitnum%3D3%26raptor%3Deagle%26pos%3Dmid%26test%3D0
[4] https://blog.akquinet.de/2020/10/09/yes-rust-has-garbage-collection-and-a-fast-one/
[5] https://www.theregister.com/2021/03/04/mobile_native_foundation/
[6] https://pubads.g.doubleclick.net/gampad/jump?co=1&iu=/6978/reg_onprem/personaltech&sz=300x50%7C300x100%7C300x250%7C300x251%7C300x252%7C300x600%7C300x601&tile=4&c=44YG3XJ2YqybwG27iTowKwlQAAABM&t=ct%3Dns%26unitnum%3D4%26raptor%3Dfalcon%26pos%3Dmid%26test%3D0
[7] https://lkml.org/lkml/2020/7/10/1261
[8] https://www.theregister.com/2020/07/13/rust_code_in_linux_kernel/
[9] https://whitepapers.theregister.com/
Re: Maybe that explains
More likely the CPU is crap, hence running "Android One" rather than the full fat thing.
Re: Android One
As far as I know Android One isn't some sort of cut-down version of the Android OS, but a program which guarantees a minimum duration of updates and near-stock Android UI. I think it was intended to cut down on "landfill Android" phones which were dirt cheap to buy but then never received any updates.
Re: Maybe that explains
Try newer Nokia feature phones, they're using KaiOS, should be what you wanted.
Re: Maybe that explains
Android ONE, 'someone' had the idea that RAM was expensive, so he spent a lot of programmer time reducing RAM consumption, so you'll find apps are constantly unloading parts and reloading parts in that OS.
It's a Google thing, I have software that has 2 activities and takes 30 seconds (2 minutes on older tablets) to recreate all its precalculation tables.
When I switch from activity 1 to activity 2, Android will unload activity 1. When I switch back, I have to wait the 30 seconds for it to finish its reload/recalc. There is 6GB of RAM free on this device.
It's dumb.
The target for Android ONE devices was 128MB of ram. RAM is a picture printed on a chip, more ram is just a slightly denser picture. How much actual price do you think they saved with that?
Re: Maybe that explains
Ugh, Symbian.
One font. GPU support left as an app-by-app problem prompting the browser with three fixed levels of zoom (and, again, rendering everything in the single Nokia font). Not POSIX compliant, weird branched dialect of C++ that looked very little like C++98, never mind having a hope of being pulled towards C++11 and subsequent. All coupled to hacked-on touch screen support.
On my Nokia N8, with no third-party software installed: three completely different kinds of text scroll area, two of them direct manipulation, one that involved dragging a scroll bar. Many, many built-in parts of the OS not yet adapted for a virtual keyboard — the process for navigating to a particular URL in the browser was this: (1) open context menu; (2) find URL entry and select it, this brings up a completely different screen with a box for typing the URL; (3) this screen isn't virtual keyboard aware, so tapping on the box brings up the full-screen keyboard. Enter your URL here and tap to enter it into the previous text box; (4) on the previous screen, tap to use what you just entered as the URL; (5) now, finally, you're returned to the browser to see your page load.
The week before the burning platforms memo I was at an official Nokia engineering event at which the sales pitch was for QtQuick, Nokia still owning Qt at the time and it being the intended isolation from Symbian's awfulness and the upward path to Maemo.
The person they'd invited — a third-party developer with a successful app — more or less presented as 'Symbian isn't that bad because with some intense coding I was able to recreate UITableView and Symbian is cool because I finally got to stick it to those designers by having the excuse of platform inability not to do most of what they wanted'. Not a convincing sales pitch.
I think the plan of killing Symbian and transitioning to Maemo via Qt was smart, it's just a shame that the unexpectedly-fast collapse of the market for feature phones in the wake of Android took away the opportunity to execute.
Garbage collection
Background history: Finalize in Java, when an object was no longer referenced, it was garbage collected, and the finalize method called, so the object could clean up anything the garbage collector doesn't know about, e.g. memory in a graphics heap, external network connections, locked resources, etc. A very useful function. A very *essential* function for a garbage collected / reference counted environment.
2010...2018, the garbage collector gets lazier and lazier, finalize cannot be relied up, it gets called less and less and later and later.
2018 Finalize gets deprecated, without a suitable replacement. Various half assed alternatives are promoted, which basically amount to the class re-creating a separate reference count and garbage collect, and forcing explicit calls to 'free' or 'close' the external resources associated with the object. Why have garbage collection then, if you need your own reference counting?? Give the class a 'FreeMeNow' destructor then if you're not prepared to fix Finalize, so at least we have one consistent reference count!!
So, now you allocate a block and *maybe* it will be freed and maybe not, so how do you check for memory leaks? Well there is a sort of way using WeakReferences. A WeakReference doesn't count as object-usage, and it goes to null when the garbage collector determines the object is no longer used by anything excepting weak references. So you could track the objects that should be freed till their weakreferences become null and you know they've been garbage collected. Bingo, it involves some polling, but there is a way.
Something like TrackLeaks(new WeakReference
So Android 8 comes and goes and this works, within 2 minutes blocks that should be freed, are freed. Leaks are found and leaks are fixed as before.
Now we get to Android 10, and I see blocks that hang around for hours. Let me give you some code so you can see how bizarre it is.
{
StringBuilder sb = new StringBuilder();
sb.add("stuff");
sb.add("stuff and more stuff");
String s1 = sb.toString();
TrackLeaks(new WeakReference
// lets clone the string to show how annoying this is
String s2 = new String(s1);
TrackLeaks(new WeakReference
return s2;
}
So, I've returned s2, and perhaps it leaks? No.... this code has been used for years, it does not leak, what's new is s1 leaks, not s2.
It's like the garbage collector is doing ONE pass, very very rarely, and finds s1 is not suitable for garbage collection because it is used by s2*. It then determines that s2 can be garbage collected and stops. Done. Except it isn't, s1 can now be
I see its a "generational heap" now in Android 10, I suspect that because s1 stays around, it is moved to the long term heap where it will hardly ever be examined for garbage collection!
I'm not sure if that's the cause, but having spent days on it, code that happily ran on Android 8, now hits the 512Mb limit on Android 10 and crashes, I am at a loss to explain these weird cases, that's my best guess.
Honestly, I'm sick of it. They have ONE JOB, manage a 8GB RAM heap for software that can happily run in 800Mb, and forces itself to run in 400Mb because of a crappy heap limit. I get max 512MB and a crappy unreliable garbage collector to work with. You give me an unstable, poorly designed OS that neither virtualizes the hardware like it should, nor delivers consistency behavior across versions like it should.
* Yes this is correct, I know the documentation says it always takes a copy, but new String(s1) takes the *value* if the string's length shows it is not a substring, so s2 would, I assume, be referencing s1 in the above case. See the code of the String constructor.
Re: Garbage collection
finalize was deprecated because it was called after an object was scheduled for collection, but as it had a hard reference to the original object it could store that reference, thus making the object not collectable. That's bad. The replacement is not WeakReference, it's PhantomReference which will be placed on a ReferenceQueue after the object has been GCed. It has no hard reference to the object, so can't resurrect it.
WeakReference doesn't "go to null when the hard references go". It goes to null if the object has been GCed. It may still point to the object, even if no other references to that object reman.
GC is about memory cleanup, not resource cleanup. If you have enough memory, the GC won't run. When your application exits, it will exit leaving some objects uncollected. Relying on the GC for cleanup of *resources* (eg temp files) is absolutely the wrong approach.
It is strange that if you mention zero initializing memory in a C or C++ project, everyone says that is inefficient and would laugh.
And yet when in a Rust project, it is absolutely fine?
I am fairly convinced that it is the mentality of a lot of C and C++ developers that needs to change, not the language.
For one, we should have had a competent address sanitizer many decades ago.
We should not be using raw pointers as much as we are. Citing that std::unique_ptr
C++ developers should also not be using raw C code without adequate bindings. Using raw, C is like the *-sys libraries in Rust. They are not the end result and should have fatter safety bindings around them.
Heck, I am also fairly convinced that iterators, .at(i) and [i] should have machinery around it to track invalidation of underlying data. At least in a debug build. C++ needs more safety, not more features. You don't even need to change the language. Something like this: https://github.com/osen/sr1
More attention needed, maybe?
" According to Google, memory-safety bugs represent 70 per cent of all high-severity security vulnerabilities found in the Android Open Source Project. "
When I was learning C, we were taught to build our own protection round hazardous functions and processes - indeed way back before the flood I co-authored an article on defeating the buffer overflow in C by use of a validation wrapper.
Size and Complexity
From what I have seen over the years is OSes and applications become bulkier as they are asked to more over time. C and C++ are fine languages for their eras particularly when the overall code bases were smaller. As the code got bulkier, it got more complex as there are more possible interactions that might be difficult to find, trace, test, etc. This makes are language like C more dangerous as it becomes more difficult for someone to properly track all the interactions and manage memory correctly particularly for some obscure interactions. This is not a criticism of C or C++ but a criticism of language developers being very slow to realize something like Rust or Go was needed a long time ago. Whether Rust has the answer or Go has a better solution; I do not expert enough to know. But this has been needed for a long time.
But if they put Rust into Android, how will a process suspend itself for a certain time period?
Because, as Neil Young reminded us all, Rust never sleeps!
[I'll show myself to the door now...]
Please Sir! My Android has gone all rusty
So old code is good and new code is bad?! I usually find it's the other way round
If uninitialized variables really are 3-5% of errors
Why didn't they change their code to initialize all variables when they are declared long ago? It would be a simple one time change to the code, plus some massaging when they add new code. I have to think a company with as many PhDs as Google could figure out a script to modify the source appropriately, or better yet modify the compiler they use to do it directly.
Just set everything to 0 if another value isn't explicitly defined. Heck, that's already true for global variables in C, the only place you need to make changes are variables declared inside functions or code blocks, and change malloc() calls to calloc() with a #define in some generic Google header that's inserted in every project.
Maybe that explains
Have a new Nokia phonie (and several very old ones). The new one runs android one. It has a collosal amount of memory but runs like a sick dog or not at all (frequently cant even put the passcode in when waking the device up!!!!!!)
Maybe this is because the Android programmers, or maybe the Nokia programmers, or maybe both are such useless twats they cant program in a proper language on a proper operating system without fucking it up.
Oh please, please, please can someone get Symbian back into circulation - my old phones with Symbian on run a LOT longer, a LOT more reliably and even with a trivial memory and processor a LOT faster. The morons who wanted to jump on the loud mouthed American inspired modern phone band wagons (apple and android) were wildly mistaken.