80 IEEE COMPUTER ARCHITECTURE LETTERS, VOL. 16, NO.

1, JANUARY-JUNE 2017

Stripes: Bit-Serial Deep Neural precision length in bits. The idea behind STR is simple: use bit-
Network Computing serial computations and compensate for the increase in computa-
tion latency by exploiting the abundant parallelism that is offered
by DNN layers. As an added benefit, using bit serial computations
Patrick Judd, Jorge Albericio, and Andreas Moshovos eliminates the need for multipliers and enables additional energy/
Abstract—The numerical representation precision required by the computations performance vs. accuracy trade offs (e.g., on a battery operated
performed by Deep Neural Networks (DNNs) varies across networks and between device a user may chose for a less accurate recognition accuracy in
layers of a same network. This observation motivates a precision-based approach exchange for longer uptime). This work demonstrates that STR has
to acceleration which takes into account both the computational structure and the the potential to improve performance with a better energy scaling
required numerical precision representation. This work presents Stripes (STR), a
and focuses solely on image classification, that is recognizing the
hardware accelerator that uses bit-serial computations to improve energy
efficiency and performance. Experimental measurements over a set of state-of-
type of object depicted in an image.
the-art DNNs for image classification show that STR improves performance over a Prior work has exploited reduced precision by turning off
state-of-the-art accelerator from 1.35
to 5.33
and by 2.24
on average. STR’s upper bit paths improving energy, but not performance [10]. STR
area and power overhead are estimated at 5 percent and 12 percent respectively. is most similar to using serial multiplication on 2D convolution to
STR is 2.00
more energy efficient than the baseline. improve energy [11]. However, STR targets 3D convolution and
does not use a lookup table of precomputed results. Since the filters
Index Terms—Hardware acceleration, deep learning, deep neural networks, in DNNs are much larger, this precomputation is intractable. Bit-
convolution, numerical representation, serial computing serial computation has been used for neural networks in [12] but
for a circuit with fixed synaptic weights.
Ç The rest of the paper is organized as follows: Section 2 corrobo-
rates the per-layer precision requirement variability of DNNs,
1 INTRODUCTION Section 3 reviews the DN design and presents Stripes, Section 4
demonstrates STR’s benefits experimentally. Finally, Section 5
DEEP neural networks (DNNs) are the state-of-the-art technique in
summarizes the limitations of our study.
many recognition tasks, spanning object [1] to speech recogni-
tion [2]. Deep Neural Networks comprise a feed-forward arrange- 2 MOTIVATION: PRECISION VARIABILITY
ment of layers each exhibiting high computational demands. They
also exhibit a high degree of parallelism which is commonly Numerical precision requirements vary significantly across net-
exploited with the use of Graphic Processing Units (GPUs). How- works and layers within a network [4], [5]. Table 1 reports the
ever, the high computation demands of DNNs and the need for fixed-point representation needed for each convolutional layer to
higher energy efficiency motivated special purpose architectures maintain the networks classification accuracy of the baseline 16-bit
such as the state-of-the-art DaDianNao (DN) [3], whose power effi- implementation. We extend the approach of Judd et al. [4], by con-
ciency is up to 330
better than a GPU. As long as additional paral- sidering a different fixed point format and including three addi-
lelism can be found, both DN and GPU performance can be tional networks [13], [14]. The precision needed varies from as
improved by introducing additional compute units. However, much as 14 bits (layer 1, GoogLeNet) to as little as 2 bits (Layer 1,
improving performance requires a proportional increase in units LeNet). We will focus on convolutional layers since they account
and thus at least a proportional increase in power. As power tends for 90 percent of the processing time in DNNs [15].
to be the limiting factor in modern high-performance designs, it is DN’s performance can be improved by 5.6
when scaling the
desirable to achieve better energy efficiency and thus performance system from 4 to 64 nodes, but with a 12.3
increase in power [3].
under given power constraints. This result demonstrates that improving performance by exploit-
This work presents Stripes (STR), a DNN performance improve- ing cross-computational parallelism can result in a disproportion-
ment technique that: 1) is complementary to existing techniques ate increase in power. Since power is the main limiting factor in
that exploit parallelism across computations, and 2) offers better modern high-performance designs, once the power budget is
energy efficiency. STR goes beyond parallelism across computa- reached, improving performance is only possible by increasing
tions and exploits the data value representation requirements of energy efficiency.
DNNs. STR is motivated by recent work that shows that the preci- Stripes exploits this precision variability to do less work per
sion required by DNNs varies significantly not only across networks neuron and thus improve energy efficiency. Compared to DN which
but also across the layers of the same network [4], [5]. Most existing uses 16-bit neurons, Stripes incorporates units whose execution time
implementations rely on a one-size-fits-all approach, using the worst- is, ideally, p=16 when using a neuron representation of p bits.
case numerical precision for all values. For example most software The Ideal column in Table 1 reports this ideal speedup over DN.
implementations use 32-bit floating-point [6], [7] while accelerators
and some recent GPUs use 16-bit fixed-point [3], [8], [9]. 3 STRIPES: A BIT-SERIAL DNN ACCELERATOR
In STR execution time scales linearly with the length of the This section first details the computation performed by convolu-
numerical precision needed by each layer. We present STR as an tional layers, and how the inner products can be transformed in
extension to the state-of-the-art accelerator DN. Since DN uses a series of additions in a straightforward way. Second, we introduce
16-bit fixed-point representation, STR would ideally improve per- the baseline system, the state-of-the-art DaDianNao accelerator [3].
formance at each layer by 16=p where p is the layer’s required Third, we present the Stripes accelerator, where the compute time
of a particular layer is directly proportional to the corresponding
 The authors are with The Edward S. Rogers Sr. Department of Electrical & Computer required precision.
Engineering, University of Toronto, Toronto, ON M5S3H7, Canada.
E-mail: [email protected], [email protected], [email protected]. 3.1 Bit-Serial Convolutional Layer Computation
Manuscript received 10 Mar. 2016; accepted 7 Apr. 2016. Date of publication 1 Aug. The input to a convolutional layer is a 3D array of neurons. The
2016; date of current version 26 June 2017. layer applies N 3D filters using a constant stride S to produce an
For information on obtaining reprints of this article, please send e-mail to: reprints@ieee.
org, and reference the Digital Object Identifier below. output 3D array of neurons. The input neuron array contains
Digital Object Identifier no. 10.1109/LCA.2016.2597140 Nx
Ny
Ni real numbers, or neurons. The layer applies Nn filters,

1556-6056 ß 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
Authorized licensed use limited to: VIT University- Chennai Campus. Downloaded on December 02,2023 at 15:14:18 UTC from IEEE Xplore. Restrictions apply.
IEEE COMPUTER ARCHITECTURE LETTERS, VOL. 16, NO. 1, JANUARY-JUNE 2017 81

TABLE 1
Per Convolutional Layer Neuron Precision Needed to
Maintain Accuracy of the Baseline

Network Neuron Precision [Bits] Ideal

LeNet 3-3 5.33
Convnet 4-8-8 2.89
AlexNet 9-8-5-5-7 2.38
NiN 8-8-8-9-7-8-8-9-9-8-8-8 1.91
GoogLeNet 10-8-10-9-8-10-9-8-9-10-7 1.76
VGG_M 7-7-7-8-7 2.23
VGG_S 7-8-9-7-9 2.04
VGG_19 12-12-12-11-12-10-11-11-13-12-13-13-13-13-13-13 1.35

Ideal: Ideal performance improvement with Stripes.

each containing Fx
Fy
Ni real numbers, or synapses. The layer

outputs a Ox
Oy
Nn neuron array (its depth equals the filter
count). The neuron arrays can be thought of comprising several fea-
tures, that is 2D arrays stacked along the i dimension, each corre-
sponding to an output feature. Applying the filter identifies where Fig. 2. DaDianNao tile.
in the input each feature appears. The input has Ni features and
the output has Nn features, each produced by a different filter. To 3.2 Baseline System
calculate an output neuron, one filter is applied over a window, a
We demonstrate STR as an extension over the DaDianNao state-of-
sub-array of the input neuron array that has the same dimensions
the-art accelerator proposed by Chen et al. [3]. A DN chip contains
as the filters Fx
Fy
Ni . Let nðy; x; iÞ and oðy; x; iÞ be respectively
16 tiles, each comprising an Neural Functional Unit (NFU), or unit,
input and output neurons, and sn ðy; x; iÞ be synapses of filter n.
and its associated buffers. Fig. 2 shows one tile. Each cycle the unit
The output neuron at position ðk; l; nÞ is calculated as in
processes 16 input neurons and 256 synapses from 16 filters, and
FX
produces 16 partial output neurons. The unit has 16 neuron lanes
y 1 FX X
x 1 N i 1
oðk; l; nÞ ¼ sn ðy; x; iÞ
nðy þ l
S; x þ k
S; iÞ: (1) and 16 filter lanes each with 16 synapse lanes (256 in total). Each
y¼0 x¼0 i¼0 neuron lane is connected to 16 synapse lanes, one from each of the
16 filter lanes. Each synapse lane multiplies its synapse with the
There is one output neuron per window and filter. The filters corresponding input neuron. The 16 synapse lanes per filter output
are applied repeatedly over different windows moving along the 16 products which are then reduced into a partial sum. Thus, each
X and Y dimensions using a constant stride S to produce all the filter lane produces a partial sum per cycle, for a total 16 output
output neurons. Accordingly, the output neuron array dimensions neurons per unit. The tile’s synapse buffer (SB) has 256 lanes
are Ox ¼ ðNx  Fx Þ=S þ 1, and Oy ¼ ðNy  Fy Þ=S þ 1. (16
16) feeding the 256 synapse lanes. The input neuron buffer
(NBin) has 16 lanes feeding the 16 neuron lanes, and the neuron
3.1.1 Bit-Serial Inner-Product Computing output buffer (NBout) has 16 lanes. The unit has 256 multipliers
Now we describe how the inner products in a DNN can be one per synapse lane and 16, 17-input adder trees one per output
computed in a serial-parallel fashion where neurons are fed in neuron (16 products plus the partial sum from NBout). The num-
bit-serially and synapses are fed in bit-parallel. The terms of ber of neuron lanes and filters per unit are design time parameters
the inner product can be reorganized as described in [16]. that could be changed. All lanes operate in lock-step.
Being nb the bth bit of n, the inner summation in Equation (1) DN is designed to minimize off-chip bandwidth and to maxi-
can be reordered as mize on-chip compute utilization. The total SB capacity is designed
to be sufficient to store all synapses for the layer(s) being processed
(32 or 2 MB per unit) thus avoiding fetching synapses from off-
X
N i 1 X
N i 1 X
P 1 X
P 1 X
N i 1
s i
ni ¼ si
nbi
2b ¼ 2b
nbi
Si : (2) chip. Up to 256 filters can be processed in parallel, 16 per unit. All
i¼0 i¼0 b¼0 b¼0 i¼0 inter-layer neuron (activation) values are stored in an appropri-
ately sized central eDRAM, or Neuron Memory (NM). The 4 MB
The synapses are first combined with the individual bits of the
NM is shared among all 16 units. The only traffic seen externally is
neurons by using a set of AND gates whose outputs are fed into an
for the initial input, for loading the synapses once per layer, and
adder tree, and then perform a shifted accumulate of the result of
for writing the final output.
the reduction. Fig. 1 shows a bit-serial inner product of two neu-
Processing starts by reading from external memory: 1) the filter
rons with two synapses.
synapses, and 2) the initial input. The filter synapses are distrib-
uted accordingly to the SBs whereas the neuron input is fed to the
NBins. Every cycle 16 neurons from a continuous slice along the i
dimension, nðx; y; iÞ; . . . ; nðx; y; i þ 15Þ, are broadcast to all units.
The layer outputs are stored through NBout to NM and then fed to
the NBins for processing the next layer. Loading the next set of syn-
apses from external memory can be overlapped with the process-
ing of the current layer as necessary. NM and the SBs are
implemented using eDRAM.

3.3 Stripes
Fig. 3 shows an STR unit that: 1) offers the same computation
throughput as a DN unit, and 2) needs p cycles to process a neuron
Fig. 1. Bit-serial inner product. represented in p bits. Since STR uses bit-serial computation, in the
Authorized licensed use limited to: VIT University- Chennai Campus. Downloaded on December 02,2023 at 15:14:18 UTC from IEEE Xplore. Restrictions apply.
82 IEEE COMPUTER ARCHITECTURE LETTERS, VOL. 16, NO. 1, JANUARY-JUNE 2017

Fig. 4. Memory mapping of elements from different windows.

reads a full NM row containing 256 neurons in parallel into a

buffer. It then broadcasts one bit from each neuron per cycle to all
units. Using two buffers, the dispatcher can in parallel read the
next group of 256 neurons.
The 256 neurons being processed by the STR unit (chunk) come
in groups of 16 neuron slices from 16 consecutive along the X
dimension windows. STR optimizes memory accesses, by storing
Fig. 3. Stripes tile. these neurons as contiguously as possible. Specifically, the input
neuron array is sliced every 16 neurons along the i-axis. For exam-
worst case, it needs 16 cycles to calculate a product involving a 16- ple, the neurons at positions ð0; 0; 15Þ and ð0; 1; 0Þ are stored contig-
bit neuron. Since a DN unit processes 16 neurons in parallel, STR uously in the same memory row. This way, one chunk fits
needs to process 16
16 or 256 neurons in parallel to maintain the perfectly in one NM row when the stride is 1. In this case, the dis-
same computation bandwidth. The natural parallelism of convolu- patcher buffer can read the whole chunk in a single cyle. However,
tional layers offers a multitude of options for processing neurons when the stride is larger, the chunk spans multiple rows and multi-
in parallel. STR opts to process 16 windows in parallel using 16 ple cycles are required to read it. If the number of rows containing
neurons from each window so that the same 16 synapses from the chunk is larger than P , the dispatcher will have to stall. The dis-
each of the 16 filters can be used to calculate 16
16 output neu- patcher incorporates a shuffling network to appropriately collect
rons in parallel. STR units process 16 neuron slices nðx; y; iÞ; . . . ; the neurons belonging to each chunk. Each neuron can come from
nðx; y; i þ 15Þ through nðx þ 15; y; iÞ; . . . ; nðx þ 15; y; i þ 15Þ bit in any of 16 different positions from the input row.
parallel, a single bit per neuron per cycle. Each neuron slice corre- Fig. 4 shows an example of how two 1
1 neuron windows with
sponds to a different window and thus the same 16 synapses can stride of two would be mapped in NM. The shuffling network
be used to calculate in parallel 256 output partial sums correspond- skips neurons ð0; 1; 0  15Þ, which are stored in between the
ing to 256 output neurons. The input neuron array of Fig. 4 shows desired neuron slices. Then the dispatcher reorganizes the chunk
an example with two windows. DN would process these windows to feed the units with bit-serial neurons.
at different cycles, whereas STR processes them in parallel bit-
serially. In both windows the same group of 16 synapses is used 4 EVALUATION
with the corresponding set of neurons to contribute a partial out-
We implemented cycle-accurate models of both the baseline sys-
put neuron, one per window.
tem and Stripes. The baseline’s area and power were measured on
Fig. 3 shows an STR unit. The unit’s front-end comprises 16
a synthesized implementation. The baseline design was imple-
groups of 16 bit-serial neuron lanes for a total of 256 neuron lanes.
mented in Verilog and synthesized via the Synopsis Design Com-
Each group of 16 neuron lanes corresponds to one of 16 input neu-
piler [17] with the TSMC 65 nm library. The NBin and NBout
ron array windows. As in DN, SB provides 16 groups of 16 16-bit
SRAM buffers were modelled using the Artisan single-ported reg-
synapses, one group per filter. While in DN each synapse is com-
ister file memory compiler [18] using double-pumping to allow a
bined with only one neuron, in STR each synapse is combined with
read and a write per cycle. The clocks of the system and SRAM
16 neurons, one from each window, allowing STR to produce 256
allow the use of this commonly used technique which estimates
partial output neurons.
the power of a dual-ported RAM. We cross-validated scaling with
Every cycle, an STR unit fetches 256 bits from NBin and each
CACTI but used the synthesis result as more representative. The
neuron bit is bit-wise ANDed with 16 synapses (one per filter) pro-
eDRAM area and energy was modelled with Destiny [19]. We have
ducing 4 K terms. The 16 per filter and per window terms are
estimated the difference in area and power between the baseline
reduced using dedicated 16-input adder trees producing in total 256
and STR with individual synthesis results for single adders and
partial output neurons per cycle. Every cycle, the partial output neu-
multipliers. A fully synthesized STR design is forthcoming. We
rons are shifted by one bit, effectively implementing bit-serial multi-
report performance for the convolutional layers of each network.
plication. The same 256 synapses are used during P cycles, where P
Power and Area The serial multiplier design in STR in combina-
is the precision, in bits, of the input neurons. After the P cycles, 256
tion with the 16
increase in parallelism results in a 72 percent
output neurons are produced in full. Stripes’s units use 256 16-input
increase in area and 87 percent increase compared to the baseline
adder trees while DN requires only 16. However, DN requires 256
parallel multipliers. In terms of the full chip, this is a 5 percent area
two-input 16-bit multipliers, whereas STR requires none.
and 12 percent power overhead.
Fig. 5 shows the speedup and relative energy efficiency for STR,
3.3.1 Neuron Memory Mapping where relative energy efficiency is calculated as the ratio
DN’s neuron memory broadcasts 16 16-bit neurons, or 256 bits per EDN =ESTR of the energy for STR and DN. On average the STR
cycle to all units. STR needs to also broadcast 256 bits per cycle to yields a speedup of 2.24
over DN. In the best case, LeNet, which
all units, where each bit corresponds to a different neuron. We opt requires only 3 bits of precision, sees a speedup of 5.33
.
to maintain the same storage format in central neuron memory VGG19 sees the least speedup, with 1.35
. This is mostly due to
(NM) as in DN aligning each neuron at a 16-bit granularity. A dis- the high precision requirements. The relative delay is in line with
patcher unit appropriately feeds the STR units. The dispatcher the ideal speedup in Table 1. The difference in these two values is
Authorized licensed use limited to: VIT University- Chennai Campus. Downloaded on December 02,2023 at 15:14:18 UTC from IEEE Xplore. Restrictions apply.
IEEE COMPUTER ARCHITECTURE LETTERS, VOL. 16, NO. 1, JANUARY-JUNE 2017 83

[7] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. B. Girshick,

S. Guadarrama, and T. Darrell, “Caffe: Convolutional architecture for fast
feature embedding,” CoRR, vol. abs/1408.5093, 2014, http://arxiv.org/
abs/1408.5093
[8] S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan, “Deep learn-
ing with limited numerical precision,” CoRR, vol. abs/1502.02551, 2015.
[9] M. Courbariaux, Y. Bengio, and J. David, “Low precision arithmetic for
deep learning,” CoRR, vol. abs/1412.7024, 2014.
[10] J. Park, J. H. Choi, and K. Roy, “Dynamic bit-width adaptation in DCT: An
approach to trade off image quality and computation energy,” IEEE Trans.
Very Large Scale Integr. Syst., vol. 18, no. 5, pp. 787–793, May 2010.
[11] T. Xanthopoulos and A. Chandrakasan, “A low-power DCT core using
adaptive bitwidth and arithmetic activity exploiting signal correlations and
quantization,” IEEE J. Solid-State Circuits, vol. 35, no. 5, pp. 740–750,
May 2000.
[12] T. Szabo, L. Antoni, G. Horvath, and B. Feher, “A full-parallel digital imple-
mentation for pre-trained NNs,” in Proc. IEEE-INNS-ENNS Int. Joint Conf.
Neural Netw., 2000, vol. 2, pp. 49–54.
[13] K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman, “Return of
the devil in the details: Delving deep into convolutional nets,” CoRR,
vol. abs/1405.3531, 2014.
[14] K. Simonyan and A. Zisserman, “Very deep convolutional networks
for large-scale image recognition,” CoRR, vol. abs/1409.1556, 2014.
[15] Y.-H. Chen, J. Emer, and V. Sze, “Eyeriss: A Spatial Architecture for
Energy-Efficient Dataflow for Convolutional Neural Networks,” 2016.
Fig. 5. Speedup and energy efficiency of STR normalized to the baseline. [Online]. Available: http://dspace.mit.edu/handle/1721.1/102369
[16] S. White, “Applications of distributed arithmetic to digital signal process-
ing: A tutorial review,” IEEE ASSP Mag., vol. 6, no. 3, pp. 4–19, Jul. 1989.
small and is due under-utilization of the neuron lanes, which is [17] Synopsys, “Design Compiler,” [Online]. Available: http://www.synopsys.
2 percent for VGG19, versus 1 percent under-utilization on average. com/Tools/Implementation/RTLSynthesis/DesignCompiler/Pages
Since there is a power overhead in STR, the relative energy effi- [18] ARM, “Artisan Memory Compiler,” [Online]. Available: http://www.arm.
com/products/physical-ip/embedded-memory-ip/index.php
ciency is lower than the speedup, but still more efficient than the [19] M. Poremba, S. Mittal, D. Li, J. Vetter, and Y. Xie, “Destiny: A tool for
baseline. The average efficiency across all networks is 2.00
, rang- modeling emerging 3D NVM and eDRAM caches,” in Proc. Des. Autom.
ing from 4.76
in the best case (LeNet) to only 1.20
in the worst Test Europe Conf. Exhibition, Mar. 2015, pp. 1543–1546.
case (VGG19).

5 STUDY LIMITATIONS
We have demonstrated the potential performance improvements
and estimated the energy efficiency characteristics of STR. The
limitations of this study include: 1) we use approximate energy
and area models. 2) We do not compare with an approach
where the data-lanes are capable of calculating different data-
types such as for example one two-input 16-bit multiplication,
or two two-input 8-bit multiplications. 3) Efficiently processing
fully-connected layers requires of a slight modification of the
present design. 4) Support for signed neuron values requires
additional hardware.
Future work will address these limitations. It is unlikely that a
design that supports only some datatypes will be competitive since
as per the results of Section 2, there aren’t many layers that would
benefit from 8-bit or 4-bit precisions. Furthermore, the unit will
have to incorporate functionality to support processing different
number of output neurons on-the-fly. We further believe that the
area and energy models we used are pessimistic. We assumed that
energy and area simply as the sum of individual adders. However,
we expect that further area and energy efficiency are to be had in
an actual design.

REFERENCES
[1] R. B. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies
for accurate object detection and semantic segmentation,” CoRR, pp. 580–
587, 2014.
[2] A. Y. Hannun, et al., “Deep speech: Scaling up end-to-end speech
recognition,” CoRR, 2014.
[3] Y. Chen, et al., “DaDianNao: A machine-learning supercomputer,” in Proc.
47th Annu. IEEE/ACM Int. Symp. Microarchitecture, Dec. 2014, pp. 609–622.
[4] P. Judd, J. Albericio, T. H. Hetherington, T. M. Aamodt, N. D. E. Jerger,
R. Urtasun, and A. Moshovos, “Reduced-precision strategies for bounded
memory in deep neural nets,” CoRR, vol. abs/1511.05236, 2015, http://
arxiv.org/abs/1511.05236
[5] S. Anwar, K. Hwang, and W. Sung, “Fixed point optimization of deep con-
volutional neural networks for object recognition,” in Proc. IEEE Int. Conf.
Acoust. Speech Signal Process., Apr. 2015, pp. 1131–1135.
[6] I. Buck, “NVIDIA’s Next-Gen Pascal GPU Architecture to Provide 10X
Speedup for Deep Learning Apps,” 2015. [Online]. Available: http://blogs.
nvidia.com/blog/2015/03/17/pascal/
Authorized licensed use limited to: VIT University- Chennai Campus. Downloaded on December 02,2023 at 15:14:18 UTC from IEEE Xplore. Restrictions apply.