Android Graphics
Jim Huang ( )
Developer & Co-Founder, 0xlab
[email protected] Sep 24, 2011 / Study-Area
Rights to copy
Copyright 2011 0xlab http://0xlab.org/
Attribution ShareAlike 3.0 Corrections, suggestions, contributions and You are free translations are welcome! to copy, distribute, display, and perform the work to make derivative works Latest update: Nov 20, 2011 to make commercial use of the work Under the following conditions Attribution. You must give the original author credit. Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under a license identical to this one. For any reuse or distribution, you must make clear to others the license terms of this work. Any of these conditions can be waived if you get permission from the copyright holder. Your fair use and other rights are in no way affected by the above. License text: http://creativecommons.org/licenses/by-sa/3.0/legalcode
[email protected]
Agenda
(1) Binder IPC (2) Android Graphics (3) 2D and Accelerations (4) OpenGL|ES
Notice: Before looking into Android Graphics, you should be aware of the design of Binder IPC, otherwise you would get lost!
Binder IPC
Processes running on Android
$ ps ... root system root root root radio root media bluetooth root keystore shell root system system app_24 radio app_18 app_7 app_0 app_14 app_3 app_25 app_26 app_27 app_1 app_19 app_21 app_28 shell $ 37 42 43 44 45 46 47 48 49 50 51 52 53 67 115 124 135 144 165 197 208 219 234 254 266 285 293 301 311 323 1 1 1 1 1 1 1 1 1 1 1 1 1 47 47 47 47 47 47 47 47 47 47 47 47 47 47 47 47 52 248 768 3824 3796 628 4336 62224 16828 1216 776 1704 696 3356 172464 80028 80732 87848 89136 86136 73996 75000 72228 85336 74656 74912 71616 72184 74728 75408 856 156 260 564 560 264 672 27576 3736 572 316 432 336 160 32596 20728 20720 20324 24160 22736 17472 18464 17652 17836 19080 18100 16280 16572 17208 18040 316 c00aef2c c022950c ffffffff ffffffff c02588c0 ffffffff c00aef2c ffffffff c00aef2c c02a8424 c02588c0 c0050934 ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff 00000000 0000875c afd0b6fc afd0bdac afd0bdac afd0c0cc afd0bdac afd0b844 afd0b6fc afd0c59c afd0b45c afd0c0cc afd0c3ac 00008294 afd0b6fc afd0c51c afd0c51c afd0c51c afd0c51c afd0c51c afd0c51c afd0c51c afd0c51c afd0c51c afd0c51c afd0c51c afd0c51c afd0c51c afd0c51c afd0c51c afd0b45c S S S S S S S S S S S S S S S S S S S S S S S S S S S S S R /sbin/ueventd /system/bin/servicemanager /system/bin/vold /system/bin/netd /system/bin/debuggerd /system/bin/rild zygote /system/bin/mediaserver /system/bin/dbus-daemon /system/bin/installd /system/bin/keystore /system/bin/sh /sbin/adbd system_server com.android.systemui com.android.inputmethod.latin com.android.phone com.android.launcher android.process.acore com.android.deskclock android.process.media com.android.bluetooth com.android.mms com.android.email com.android.providers.calendar com.android.protips com.android.music com.android.quicksearchbox com.cooliris.media ps
More than 30 processes (200+ threads).
6
IPC = Inter-Process Communication
Activity Activity Manager Window Manager Alarm Manager
Kernel
IPC Abstraction
Intent AIDL
More abstract
Intent The highest level abstraction Inter process method invocation AIDL: Android Interface Definition Language binder: kernel driver ashmem: shared memory
Binder
Method invocation
caller
callee
In the same process
Inter-process method invocation
caller
interface
caller
interface
How?
callee
interface
callee
10
Inter-process method invocation
caller
interface
caller
interface
Proxy
Binder in kernel
Binder Thread
callee
Stub callee
interface
11
android.os.Parcel
Delivering arguments of method
flatten
unflatten
transmit
12
UML Representation
<<interface>> implements
Proxy
Stub
13
UML Representation
caller
<<interface>> implements
calls
Proxy
Stub extends callee
14
AIDL
Auto generated from .aidl file
caller
<<interface>>
Proxy
Stub
callee
15
Use Case: Who calls onPause() in Activity?
2
Send message by Handler
Activity
3
queue Activity Manager Binder Thread #1
Looper
OnPause() is called in main thread
Main Thread
Kernel
Call schedulePauseActivity across process
16
Binder Multi-thread aware
Have internal status per thead Compare to UNIX socket: sockets have internal status per file descriptor (FD)
17
Binder
A pool of threads is associated to each service application to process incoming IPC (Inter-Process Communication). Binder performs mapping of object between two processes. Binder uses an object reference as an address in a processs memory space. Synchronous call, reference couting
Binder
Binder is different from UNIX socket
socket
internal status associated to FD
binder
associated to PID
(FD can be shared among threads in the same process)
read & write operation network transparency
stream I/O Yes
done at once by ioctl No
expected local only
20
Transaction of Binder
binder_write_read write_size write_consumed write_buffer read_size read_consumed read_buffer read buffer write buffer
if (ioctl(fd, BINDER_WRITE_READ, &bwt ) >= 0) err = NO_ERROR; else err = -errno;
21
Transaction of Binder
Process A and B have different memory space. They can not see each other. Kernel Binder Process B Process A
Copy memory by copy_from _user Then, wake up process B Kernel Binder Process B
Process A Copy memory by copy_to_user
Internally, Android uses Binder for graphics data transaction across processes. It is fairly efficient.
22
Real Case
Binder IPC is used for communicating between Graphics client and server. Taken from http://www.cnblogs.com/xl19862005/archive/2011/11/17/2215363.html
23
Android / Anonymous SHared MEMory subsystem
system/core/cutils/ashmem.h
int ashmem_create_region(const char *name, size_t size) int ashmem_set_prot_region(int fd, int prot) int ashmem_pin_region(int fd, size_t offset, size_t len) int ashmem_unpin_region(int fd, size_t offset, size_t len)
Ashmem
a named memory block shared between processes that the kernel is allowed to free. This is notable as the kernel is not allowed to free standard shared memory. Similar to weak reference of Java. Useful to implement cache. Used in android.os.MemoryFile (Java), 2D memory allocator, etc.
24
Android Graphics
Surface
Source: frameworks/base/core/java/android/view/Surface.java
/* Handle on to a raw buffer that is being managed by the screen compositor */ public class Surface implements Parcelable { public Surface() {
mCanvas = new CompatibleCanvas();
} private class CompatibleCanvas extends Canvas { /* ... */ } }
Surface instances can be written to and restored from a Parcel. Surface instances can be written to and restored from a Parcel.
Delivering arguments of method
flatten
unflatten
transmit
Properties
Android SurfaceFlinger
Can combine 2D/3D surfaces and surfaces from multiple applications Surfaces passed as buffers via Binder IPC calls Can use OpenGL ES and 2D hardware accelerator for its compositions
Double-buffering using page-flip
from SurfaceFlinger to Framebuffer
PixelFlinger : software renderer
Render functions: pointx, linex, recti, trianglex Texture and color buffer: activeTexture, bindTexture,
colorBuffer, readBuffer, depthBuffer, BindTextureLod
Device framebuffer functions: copyPixels,
rasterPos2x, rasterPos2i
Optimizer: codeflinger (JIT assembler)
I/SurfaceFlinger(1931):OpenGLinformations: I/SurfaceFlinger(1931):vendor:Android I/SurfaceFlinger(1931):renderer:AndroidPixelFlinger1.2 I/SurfaceFlinger(1931):version:OpenGLESCM1.0
GGLAssembler GGLAssembler codeflinger's JIT Assembler codeflinger's JIT Assembler
2D and Accelerator
Skia
SkBitmap SkBitmap::Allocator GPU integration in Android 4.0
Case Study: WebKit rendering
Fill rectangle FillRectangles()
Set clip region Acquire font cache Show glyphs BatchBlit() Allocate font cache Surface composite Blit() TileBlit() Prepare / finish composite Clear() Blit(), memcpy()
2D Compositing
SRC
SRC & DST
1 -asrc
DST
Compositing
SRC & NO-DST
NO-DST & NO-SRC 1-adst adst
Enable 2D Accelerator
Android Graphics (HAL view)
Android Framework (Java) libandroid_runtime EventHub Surfaceflinger (service) Copybit (HW accelerated)
libui
LibGLES (libagl)
libpixelflinger
2D Accelerator for Android Graphics
libcopybit provides hareware bitblit operations which includes moving, scaling, rotation, mirroring, and more effects, like blending, dithering, bluring, etc. Removed since Android 2.3 Android has two copybit interfaces: Blit: moving / blending Stretch: scaling besides moving libcopybit is called by libagl which can do swapBuffers to do the framebuffer page flipping that can also be accelerated by libcopybit.
Copybit could improve the performance of page flipping Copybit could improve the performance of page flipping
Copybit operations
Copybit: 2D blitter Copybit: 2D blitter
PMEM:
Manipulate physically continuous memory
Hardware graphics/bitblt operations (blitter) needs physical continuous memory to manipulate. Android use libgralloc to allocate pmem (physical continuous memory) for android native buffer. pmem driver can support up to 12 devices, we have only one for copybit (Android, android native buffer) While running 0xbench, the peak size of the pmem allocated (mapped) is 25194496 bytes.
Take Qualcomm MSM7x25 for example: Take Qualcomm MSM7x25 for example: /dev/pmem /dev/pmem /dev/pmem_adsp /dev/pmem_adsp For multimedia codec, audio, video, camera For multimedia codec, audio, video, camera
gralloc & pmem
gralloc_alloc
Offset = Alloc(size), PMEM_CONNECT, PMEM_MAP, memset If success, new private_handle_t(fd, size, flags); gralloc module
gralloc_free
PMEM_UNMAP, deallocate(offset)
pmem memory
(mmaped virtual address)base 0x49422336, c06c4000, 6c4000 buffer base = base + offset Buffer 2 master_fd (mmap size: 47MB) Buffer 1 fd when being freed, this will be closed
Buffer 3
Quantum
Debug pmem, /sys/kernel/pmem_regions
pmem pmem can have up to 12 devices..., we have only one for copybit (Android native buffer)
gralloc/copybit structure
android_native_buffer_t buffer_handle_t handle; copybit_image_t native_handle_t* handle;
native_handle_t
private_handle_t
Fd (pmem dev) Offset Size Base Flags lockState
buffer_handle_t
libgralloc
virtual address
Android native buffer contains buffer_handle_t Which is native_handle * , defined in system/cutils/
private_handle_t::PRIV_FLAGS_U SES_PMEM
private_handle_t::LOCK_STATE_MAPPED
Functions in copybit HAL
BLIT (moving) Stretch (scaling) Alpha-blending Unused in real case Rotate Unused in real case
OpenGL|ES
Key Concepts
Android is moving to OpenGL|ES accelerated rendering since version 2.x Window systems already comprehend z-order 3D != 2D with depth OpenGL is object based
Describes a scene using its components and properties. Output quality is dependent on the renderer, not source
OpenGL Terminology
OpenGL
An API from Khronos (from SGI), for constructing a 3D object, doing operations on the object, and displaying it
Primitives
Triangles, Lines, Points, that can be specified through vertices to define an arbitrary shape
Texture
Small (!) bitmap to make objects more realistic
EGL
The EGL API defines a portable mechanism for creating GL contexts and windows for rendering into, which may be used in conjunction with different native platform window systems using the WSEGL layer
EGL
EGL (Embedded-System Graphics Library) is an interface between Khronos rendering APIs (such as OpenGL ES or OpenVG) and the underlying native platform window system. EGL handles graphics context management, surface/buffer binding, and rendering synchronization and enables high-performance, accelerated, mixedmode 2D and 3D rendering using other Khronos APIs. EGL Surfaces windows - on-screen rendering pbuffers - off-screen rendering pixmaps - off-screen rendering
from EGL to SurfaceFlinger
hgl = hardware hgl = hardware OpenGL|ES OpenGL|ES
agl = android software agl = android software OpenGL|ES renderer OpenGL|ES renderer
http://0xlab.org
