mistress
stb_image_resize2.h
1/* stb_image_resize2 - v2.50 - public domain image resizing
2
3 by Jeff Roberts (v2) and Jorge L Rodriguez
4 http://github.com/nothings/stb
5
6 Can be threaded with the extended API. SSE2, AVX, Neon and WASM SIMD support. Only
7 scaling and translation is supported, no rotations or shears.
8
9 COMPILING & LINKING
10 In one C/C++ file that #includes this file, do this:
11 #define STB_IMAGE_RESIZE_IMPLEMENTATION
12 before the #include. That will create the implementation in that file.
13
14 EASY API CALLS:
15 Easy API downsamples w/Mitchell filter, upsamples w/cubic interpolation, clamps to edge.
16
17 stbir_resize_uint8_srgb( input_pixels, input_w, input_h, input_stride_in_bytes,
18 output_pixels, output_w, output_h, output_stride_in_bytes,
19 pixel_layout_enum )
20
21 stbir_resize_uint8_linear( input_pixels, input_w, input_h, input_stride_in_bytes,
22 output_pixels, output_w, output_h, output_stride_in_bytes,
23 pixel_layout_enum )
24
25 stbir_resize_float_linear( input_pixels, input_w, input_h, input_stride_in_bytes,
26 output_pixels, output_w, output_h, output_stride_in_bytes,
27 pixel_layout_enum )
28
29 If you pass NULL or zero for the output_pixels, we will allocate the output buffer
30 for you and return it from the function (free with free() or STBIR_FREE).
31 As a special case, XX_stride_in_bytes of 0 means packed continuously in memory.
32
33 API LEVELS
34 There are three levels of API - easy-to-use, medium-complexity and extended-complexity.
35
36 See the "header file" section of the source for API documentation.
37
38 ADDITIONAL DOCUMENTATION
39
40 MEMORY ALLOCATION
41 By default, we use malloc and free for memory allocation. To override the
42 memory allocation, before the implementation #include, add a:
43
44 #define STBIR_MALLOC(size,user_data) ...
45 #define STBIR_FREE(ptr,user_data) ...
46
47 Each resize makes exactly one call to malloc/free (unless you use the
48 extended API where you can do one allocation for many resizes). Under
49 address sanitizer, we do separate allocations to find overread/writes.
50
51 PERFORMANCE
52 This library was written with an emphasis on performance. When testing
53 stb_image_resize with RGBA, the fastest mode is STBIR_4CHANNEL with
54 STBIR_TYPE_UINT8 pixels and CLAMPed edges (which is what many other resize
55 libs do by default). Also, make sure SIMD is turned on of course (default
56 for 64-bit targets). Avoid WRAP edge mode if you want the fastest speed.
57
58 This library also comes with profiling built-in. If you define STBIR_PROFILE,
59 you can use the advanced API and get low-level profiling information by
60 calling stbir_resize_extended_profile_info() or stbir_resize_split_profile_info()
61 after a resize.
62
63 SIMD
64 Most of the routines have optimized SSE2, AVX, AVX-512, NEON and WASM versions.
65
66 On Microsoft compilers, we automatically turn on SIMD for 64-bit x64 and
67 ARM; for 32-bit x86 and ARM, you select SIMD mode by defining STBIR_SSE2 or
68 STBIR_NEON.
69
70 For AVX and AVX2, we auto-select it by detecting the /arch:AVX
71 or /arch:AVX2 switches. You can also always manually turn SSE2, AVX or AVX2
72 support on by defining STBIR_SSE2, STBIR_AVX or STBIR_AVX2.
73
74 For AVX-512, we turn it on by /arch:AVX512 (or -march=x86-64-v4, or the
75 -mavx512f -mavx512bw -mavx512vl trio on clang and GCC), or by defining
76 STBIR_AVX512. It's mostly useful on Zen 5 or modern Intel cores that don't
77 downclock on use. Best on very large resizes.
78
79 On Linux, SSE2 and Neon is on by default for 64-bit x64 or ARM64. For 32-bit,
80 we select x86 SIMD mode by whether you have -msse2, -mavx or -mavx2 enabled
81 on the command line. For 32-bit ARM, you must pass -mfpu=neon-vfpv4 for both
82 clang and GCC, but GCC also requires an additional -mfp16-format=ieee to
83 automatically enable NEON.
84
85 On x86 platforms, you can also define STBIR_FP16C to turn on FP16C instructions
86 for converting back and forth to half-floats. This is autoselected when we
87 are using AVX2. Clang and GCC also require the -mf16c switch. ARM always uses
88 the built-in half float hardware NEON instructions.
89
90 You can also tell us to use multiply-add instructions with STBIR_USE_FMA.
91 Because x86 doesn't always have fma, we turn it off by default to maintain
92 determinism across all platforms. If you don't care about non-FMA determinism
93 and are willing to restrict yourself to more recent x86 CPUs (around the AVX
94 timeframe), then fma will give you around a 15% speedup.
95
96 You can force off SIMD in all cases by defining STBIR_NO_SIMD. You can turn
97 off AVX, AVX2 or AVX-512 specifically with STBIR_NO_AVX, STBIR_NO_AVX2 or
98 STBIR_NO_AVX512. AVX is 10% to 40% faster, and AVX2 is generally another 12%.
99
100 ALPHA CHANNEL
101 Most of the resizing functions provide the ability to control how the alpha
102 channel of an image is processed.
103
104 When alpha represents transparency, it is important that when combining
105 colors with filtering, the pixels should not be treated equally; they
106 should use a weighted average based on their alpha values. For example,
107 if a pixel is 1% opaque bright green and another pixel is 99% opaque
108 black and you average them, the average will be 50% opaque, but the
109 unweighted average and will be a middling green color, while the weighted
110 average will be nearly black. This means the unweighted version introduced
111 green energy that didn't exist in the source image.
112
113 (If you want to know why this makes sense, you can work out the math for
114 the following: consider what happens if you alpha composite a source image
115 over a fixed color and then average the output, vs. if you average the
116 source image pixels and then composite that over the same fixed color.
117 Only the weighted average produces the same result as the ground truth
118 composite-then-average result.)
119
120 Therefore, it is in general best to "alpha weight" the pixels when applying
121 filters to them. This essentially means multiplying the colors by the alpha
122 values before combining them, and then dividing by the alpha value at the
123 end.
124
125 The computer graphics industry introduced a technique called "premultiplied
126 alpha" or "associated alpha" in which image colors are stored in image files
127 already multiplied by their alpha. This saves some math when compositing,
128 and also avoids the need to divide by the alpha at the end (which is quite
129 inefficient). However, while premultiplied alpha is common in the movie CGI
130 industry, it is not commonplace in other industries like videogames, and most
131 consumer file formats are generally expected to contain not-premultiplied
132 colors. For example, Photoshop saves PNG files "unpremultiplied", and web
133 browsers like Chrome and Firefox expect PNG images to be unpremultiplied.
134
135 Note that there are three possibilities that might describe your image
136 and resize expectation:
137
138 1. images are not premultiplied, alpha weighting is desired
139 2. images are not premultiplied, alpha weighting is not desired
140 3. images are premultiplied
141
142 Both case #2 and case #3 require the exact same math: no alpha weighting
143 should be applied or removed. Only case 1 requires extra math operations;
144 the other two cases can be handled identically.
145
146 stb_image_resize expects case #1 by default, applying alpha weighting to
147 images, expecting the input images to be unpremultiplied. This is what the
148 COLOR+ALPHA buffer types tell the resizer to do.
149
150 When you use the pixel layouts STBIR_RGBA, STBIR_BGRA, STBIR_ARGB,
151 STBIR_ABGR, STBIR_RA, or STBIR_AR you are telling us that the pixels are
152 non-premultiplied. In these cases, the resizer will alpha weight the colors
153 (effectively creating the premultiplied image), do the filtering, and then
154 convert back to non-premult on exit.
155
156 When you use the pixel layouts STBIR_RGBA_PM, STBIR_BGRA_PM, STBIR_ARGB_PM,
157 STBIR_ABGR_PM, STBIR_RA_PM or STBIR_AR_PM, you are telling that the pixels
158 ARE premultiplied. In this case, the resizer doesn't have to do the
159 premultipling - it can filter directly on the input. This about twice as
160 fast as the non-premultiplied case, so it's the right option if your data is
161 already setup correctly.
162
163 When you use the pixel layout STBIR_4CHANNEL or STBIR_2CHANNEL, you are
164 telling us that there is no channel that represents transparency; it may be
165 RGB and some unrelated fourth channel that has been stored in the alpha
166 channel, but it is actually not alpha. No special processing will be
167 performed.
168
169 The difference between the generic 4 or 2 channel layouts, and the
170 specialized _PM versions is with the _PM versions you are telling us that
171 the data *is* alpha, just don't premultiply it. That's important when
172 using SRGB pixel formats, we need to know where the alpha is, because
173 it is converted linearly (rather than with the SRGB converters).
174
175 Because alpha weighting produces the same effect as premultiplying, you
176 even have the option with non-premultiplied inputs to let the resizer
177 produce a premultiplied output. Because the intially computed alpha-weighted
178 output image is effectively premultiplied, this is actually more performant
179 than the normal path which un-premultiplies the output image as a final step.
180
181 Finally, when converting both in and out of non-premulitplied space (for
182 example, when using STBIR_RGBA), we go to somewhat heroic measures to
183 ensure that areas with zero alpha value pixels get something reasonable
184 in the RGB values. If you don't care about the RGB values of zero alpha
185 pixels, you can call the stbir_set_non_pm_alpha_speed_over_quality()
186 function - this runs a premultiplied resize about 25% faster. That said,
187 when you really care about speed, using premultiplied pixels for both in
188 and out (STBIR_RGBA_PM, etc) much faster than both of these premultiplied
189 options.
190
191 PIXEL LAYOUT CONVERSION
192 The resizer can convert from some pixel layouts to others. When using the
193 stbir_set_pixel_layouts(), you can, for example, specify STBIR_RGBA
194 on input, and STBIR_ARGB on output, and it will re-organize the channels
195 during the resize. Currently, you can only convert between two pixel
196 layouts with the same number of channels.
197
198 DETERMINISM
199 We commit to being deterministic (from x64 to ARM to scalar to SIMD, etc).
200 This requires compiling with fast-math off (using at least /fp:precise).
201 Also, you must turn off fp-contracting (which turns mult+adds into fmas)!
202 We attempt to do this with pragmas, but with Clang, you usually want to add
203 -ffp-contract=off to the command line as well.
204
205 For 32-bit x86, you must use SSE and SSE2 codegen for determinism. That is,
206 if the scalar x87 unit gets used at all, we immediately lose determinism.
207 On Microsoft Visual Studio 2008 and earlier, from what we can tell there is
208 no way to be deterministic in 32-bit x86 (some x87 always leaks in, even
209 with fp:strict). On 32-bit x86 GCC, determinism requires both -msse2 and
210 -fpmath=sse.
211
212 Note that we will not be deterministic with float data containing NaNs -
213 the NaNs will propagate differently on different SIMD and platforms.
214
215 If you turn on STBIR_USE_FMA, then we will be deterministic with other
216 fma targets, but we will differ from non-fma targets (this is unavoidable,
217 because a fma isn't simply an add with a mult - it also introduces a
218 rounding difference compared to non-fma instruction sequences.
219
220 FLOAT PIXEL FORMAT RANGE
221 Any range of values can be used for the non-alpha float data that you pass
222 in (0 to 1, -1 to 1, whatever). However, if you are inputting float values
223 but *outputting* bytes or shorts, you must use a range of 0 to 1 so that we
224 scale back properly. The alpha channel must also be 0 to 1 for any format
225 that does premultiplication prior to resizing.
226
227 Note also that with float output, using filters with negative lobes, the
228 output filtered values might go slightly out of range. You can define
229 STBIR_FLOAT_LOW_CLAMP and/or STBIR_FLOAT_HIGH_CLAMP to specify the range
230 to clamp to on output, if that's important.
231
232 MAX/MIN SCALE FACTORS
233 Images can be up to 2^30 pixels wide or high (dimensions are ints, and
234 the internal pointer math is done in size_t).
235
236 The positions of the filter taps are calculated in stbir_real (double by
237 default), which is accurate all the way to that limit. When the scale
238 works out to an exact fraction (which it does for most whole image
239 resizes), the positions are calculated with 64-bit integer math and are
240 exact at any size.
241
242 If you are on a part without hardware doubles, you can define stbir_real
243 to float before you #include. Positions are then accurate to about 2^20
244 pixels and are only off by half a pixel at 2^24, so for extreme resizes
245 on those parts, do it in multiple stages (using float intermediate
246 buffers).
247
248 HUGE IMAGES AND TILING
249 Image strides are 64-bit, so the buffers themselves can be as large as
250 memory allows; widths and heights are limited to 2^30 (ints). To resize
251 an image larger than that in either direction, resize it in tiles: for
252 each output tile, point the resize at a piece of the input (the 64-bit
253 strides reach anywhere in the buffer) with the output tile's size, and
254 set the input region the tile maps to, in the piece's own coordinates,
255 with stbir_set_input_subrect. Two things keep the tiles seamless:
256
257 - the piece must extend past the region by the filter's reach on every
258 side that isn't a real image edge (the filter's support in input
259 pixels: support x input / output when downsampling, support when
260 upsampling, plus a pixel of slack), because the edge modes act at
261 the piece's edges - a tap that runs off the piece is clamped or
262 wrapped there, as if the image ended. WRAP is the awkward one: a
263 tile at a real edge needs the pixels from the opposite edge of the
264 whole image, which a piece of the buffer can't reach, so you would
265 have to copy them into place next to the piece - it is simpler to
266 use another edge mode when tiling.
267
268 - the region must be exactly the tile's share of the whole mapping
269 (tile size x whole input / whole output, at the tile's offset), so
270 every tile has the whole resize's scale and phase - a region drawn
271 from rounded numbers shifts that tile's samples against its
272 neighbors.
273
274 A region that reaches past the source is clipped to it (the output
275 pixels that would map outside are not written).
276
277 FLIPPED IMAGES
278 Stride is just the delta from one scanline to the next. This means you can
279 use a negative stride to handle inverted images (point to the final
280 scanline and use a negative stride). You can invert the input or output,
281 using negative strides.
282
283 DEFAULT FILTERS
284 For functions which don't provide explicit control over what filters to
285 use, you can change the compile-time defaults with:
286
287 #define STBIR_DEFAULT_FILTER_UPSAMPLE STBIR_FILTER_something
288 #define STBIR_DEFAULT_FILTER_DOWNSAMPLE STBIR_FILTER_something
289
290 See stbir_filter in the header-file section for the list of filters.
291
292 NEW FILTERS
293 A number of 1D filter kernels are supplied. For a list of supported
294 filters, see the stbir_filter enum. You can install your own filters by
295 using the stbir_set_filter_callbacks_new function.
296
297 PROGRESS
298 For interactive use with slow resize operations, you can use the
299 scanline callbacks in the extended API. It would have to be a *very* large
300 image resample to need progress though - we're very fast.
301
302 ASSERT
303 Define STBIR_ASSERT(boolval) to override assert() and not use assert.h
304
305 CACHE SIZE
306 A resize is run in column slices sized so that its working memory fits
307 the L2 cache. By default, the L2 size is read with cpuid on x86 (and a L2
308 of a megabyte or more is treated as 512 KB, to keep the internal memory
309 down), and taken as 256 KB on other cpus. If that isn't right for you,
310 set stbir_l2_cache_size_override (bytes) before resizing, and it is used
311 as is; or define STBIR_ASK_WINDOWS_FOR_L2_SIZE, STBIR_ASK_LINUX_FOR_L2_SIZE
312 or STBIR_ASK_MACOS_FOR_L2_SIZE before the implementation include to ask
313 the OS instead of cpuid (see the comment above them for the headers each
314 one needs). Usually the defaults will just work fine.
315
316 THREADING
317 To resize on several threads, call stbir_build_samplers_with_splits on
318 one thread with the number of threads you'd like, then call
319 stbir_resize_extended_split on each thread with its index (and a count
320 of 1), and stbir_free_samplers when they're all done. The build returns
321 the number of splits to actually use: it is limited to how many ways
322 the resize can be split, and then to the count the build estimates
323 will still speed things up (a small image or very narrow rows don't
324 get faster with more threads, they get slower). The estimate's constants
325 are statics in the implementation that you can override if you need to.
326
327 FUTURE TODOS
328 * Add pixel layout conversions for sensible different channel counts
329 (maybe, 1->3/4, 3->4, 4->1, 3->1).
330 * Convert the reincludes to macros when we know they aren't changing.
331 * Most of our code is internally function pointers, should we compile
332 all the SIMD stuff always and dynamically dispatch?
333
334 CONTRIBUTORS
335 Jeff Roberts: 2.0 implementation, optimizations, SIMD
336 Martins Mozeiko: NEON simd, WASM simd, clang and GCC whisperer
337 Fabian Giesen: half float and srgb converters
338 Sean Barrett: API design, optimizations
339 Jorge L Rodriguez: Original 1.0 implementation
340 Aras Pranckevicius: bugfixes
341 Nathan Reed: warning fixes for 1.0
342
343 REVISIONS
344 2.50 (2026-10-07) Big performance release, especially for large and/or
345 threaded resizes! New stuff:
346 - Resizes now run in column slices sized to the L2 cache
347 (auto-detected, or set stbir_l2_cache_size_override),
348 which is 1.6-3x faster on large resizes (more when
349 threaded) and uses far less temp memory.
350 - Coefficient sets for rational scales are now shared,
351 using much less memory for these resizes (and faster).
352 - Rational downsamples now get faster dedicated kernels
353 (20-30% faster).
354 - Added horizontal gather kernels up to 17 taps, so
355 4-tap filters stay on the fast gather path down to
356 4:1 (20-30% faster than scatter path). Useful in mip
357 chain generation loops.
358 - For each resize, we adjust the offset of the internal
359 buffers to avoid 4K aliasing with the input buffer
360 (avoids random 40% slowdowns).
361 - New model for estimating vertical or horizontal first
362 mode. Much better at predicting best direction (lots
363 of resizes faster 30%-40% with this change).
364 - Added a model of thread addition improvement - if you
365 ask for 32 threads for example, the model will only use
366 the thread count that will improve performance (some
367 users were estimating this themselves - this should
368 be more accurate).
369 - 1 channel downsizes 30-40% faster (SIMD across samples).
370 - Added some new AVX kernels and optimizations.
371 - Added AVX-512 support - useful mostly just for Zen 5
372 and recent Intel cores and large resamples.
373 - Bunch of cleanup and exactness in coefficient building,
374 along with new higher precision filter callbacks - this
375 moves our image resize limits up to 2^30.
376 - Added several new filters - more Mitchell variants,
377 some Lanczos, and the Muratori filter.
378 - Point sampling now actually does point sampling when
379 downsampling (previously only upsamples were point
380 sampled, down was averaged).
381 - Added 64-bit strides so that you can resize over the
382 2^30 with manual tiling (calling resize multiple
383 times over the 64-bit area).
384 - Windows ARM64EC support (thanks navvyswethgraphics)
385 - Lots of little bugfixes in unusual corner cases.
386 NOTE THAT 2.50 IS NOT deterministic with prior versions
387 (among other improvements: coefficients calculated more
388 precisely, new v-first vs h-first decisions, etc).
389
390 2.18 (2026-03-25) fixed coefficient calculation when skipping a coefficient off
391 the left side of the window, added non-aligned access safe
392 memcpy mode for scalar path, fixed various typos, and fixed
393 define error in the float clamp output mode.
394 2.17 (2025-10-25) silly format bug in easy-to-use APIs.
395 2.16 (2025-10-21) fixed the easy-to-use APIs to allow inverted bitmaps (negative
396 strides), fix vertical filter kernel callback, fix threaded
397 gather buffer priming (and assert).
398 (thanks adipose, TainZerL, and Harrison Green)
399 2.15 (2025-07-17) fixed an assert in debug mode when using floats with input
400 callbacks, work around GCC warning when adding to null ptr
401 (thanks Johannes Spohr and Pyry Kovanen).
402 2.14 (2025-05-09) fixed a bug using downsampling gather horizontal first, and
403 scatter with vertical first.
404 2.13 (2025-02-27) fixed a bug when using input callbacks, turned off simd for
405 tiny-c, fixed some variables that should have been static,
406 fixes a bug when calculating temp memory with resizes that
407 exceed 2GB of temp memory (very large resizes).
408 2.12 (2024-10-18) fix incorrect use of user_data with STBIR_FREE
409 2.11 (2024-09-08) fix harmless asan warnings in 2-channel and 3-channel mode
410 with AVX-2, fix some weird scaling edge conditions with
411 point sample mode.
412 2.10 (2024-07-27) fix the defines GCC and mingw for loop unroll control,
413 fix MSVC 32-bit arm half float routines.
414 2.09 (2024-06-19) fix the defines for 32-bit ARM GCC builds (was selecting
415 hardware half floats).
416 2.08 (2024-06-10) fix for RGB->BGR three channel flips and add SIMD (thanks
417 to Ryan Salsbury), fix for sub-rect resizes, use the
418 pragmas to control unrolling when they are available.
419 2.07 (2024-05-24) fix for slow final split during threaded conversions of very
420 wide scanlines when downsampling (caused by extra input
421 converting), fix for wide scanline resamples with many
422 splits (int overflow), fix GCC warning.
423 2.06 (2024-02-10) fix for identical width/height 3x or more down-scaling
424 undersampling a single row on rare resize ratios (about 1%).
425 2.05 (2024-02-07) fix for 2 pixel to 1 pixel resizes with wrap (thanks Aras),
426 fix for output callback (thanks Julien Koenen).
427 2.04 (2023-11-17) fix for rare AVX bug, shadowed symbol (thanks Nikola Smiljanic).
428 2.03 (2023-11-01) ASAN and TSAN warnings fixed, minor tweaks.
429 2.00 (2023-10-10) mostly new source: new api, optimizations, simd, vertical-first, etc
430 2x-5x faster without simd, 4x-12x faster with simd,
431 in some cases, 20x to 40x faster esp resizing large to very small.
432 0.96 (2019-03-04) fixed warnings
433 0.95 (2017-07-23) fixed warnings
434 0.94 (2017-03-18) fixed warnings
435 0.93 (2017-03-03) fixed bug with certain combinations of heights
436 0.92 (2017-01-02) fix integer overflow on large (>2GB) images
437 0.91 (2016-04-02) fix warnings; fix handling of subpixel regions
438 0.90 (2014-09-17) first released version
439
440 LICENSE
441 See end of file for license information.
442*/
443
444#if !defined(STB_IMAGE_RESIZE_DO_HORIZONTALS) && !defined(STB_IMAGE_RESIZE_DO_VERTICALS) && !defined(STB_IMAGE_RESIZE_DO_CODERS) // for internal re-includes
445
446#ifndef STBIR_INCLUDE_STB_IMAGE_RESIZE2_H
447#define STBIR_INCLUDE_STB_IMAGE_RESIZE2_H
448
449#include <stddef.h>
450#ifdef _MSC_VER
451typedef unsigned char stbir_uint8;
452typedef unsigned short stbir_uint16;
453typedef unsigned int stbir_uint32;
454typedef unsigned __int64 stbir_uint64;
455typedef __int64 stbir_int64;
456#else
457#include <stdint.h>
458typedef uint8_t stbir_uint8;
459typedef uint16_t stbir_uint16;
460typedef uint32_t stbir_uint32;
461typedef uint64_t stbir_uint64;
462typedef int64_t stbir_int64;
463#endif
464
465// we use this type for high precision floating point work - it's normally just a double
466// but you can define it to float, if you don't have double support (some old embedded
467// platforms).
468#ifndef stbir_real
469#define stbir_real double
470#endif
471
472#ifndef STBIRDEF
473#ifdef STB_IMAGE_RESIZE_STATIC
474#define STBIRDEF static
475#else
476#ifdef __cplusplus
477#define STBIRDEF extern "C"
478#else
479#define STBIRDEF extern
480#endif
481#endif
482#endif
483
484//////////////////////////////////////////////////////////////////////////////
485//// start "header file" ///////////////////////////////////////////////////
486//
487// Easy-to-use API:
488//
489// * stride is the offset between successive rows of image data
490// in memory, in bytes (64-bit, so a buffer can be larger than 2 GB).
491// specify 0 for packed continuously in memory
492// * colorspace is linear or sRGB as specified by function name
493// * Uses the default filters
494// * Uses edge mode clamped
495// * returned result is 1 for success or 0 in case of an error.
496
497
498// stbir_pixel_layout specifies:
499// number of channels
500// order of channels
501// whether color is premultiplied by alpha
502// for back compatibility, you can cast the old channel count to an stbir_pixel_layout
503typedef enum
504{
505 STBIR_1CHANNEL = 1,
506 STBIR_2CHANNEL = 2,
507 STBIR_RGB = 3, // 3-chan, with order specified (for channel flipping)
508 STBIR_BGR = 0, // 3-chan, with order specified (for channel flipping)
509 STBIR_4CHANNEL = 5,
510
511 STBIR_RGBA = 4, // alpha formats, where alpha is NOT premultiplied into color channels
512 STBIR_BGRA = 6,
513 STBIR_ARGB = 7,
514 STBIR_ABGR = 8,
515 STBIR_RA = 9,
516 STBIR_AR = 10,
517
518 STBIR_RGBA_PM = 11, // alpha formats, where alpha is premultiplied into color channels
519 STBIR_BGRA_PM = 12,
520 STBIR_ARGB_PM = 13,
521 STBIR_ABGR_PM = 14,
522 STBIR_RA_PM = 15,
523 STBIR_AR_PM = 16,
524
525 STBIR_RGBA_NO_AW = 11, // alpha formats, where NO alpha weighting is applied at all!
526 STBIR_BGRA_NO_AW = 12, // these are just synonyms for the _PM flags (which also do
527 STBIR_ARGB_NO_AW = 13, // no alpha weighting). These names just make it more clear
528 STBIR_ABGR_NO_AW = 14, // for some folks).
529 STBIR_RA_NO_AW = 15,
530 STBIR_AR_NO_AW = 16,
531
532} stbir_pixel_layout;
533
534//===============================================================
535// Simple-complexity API
536//
537// If output_pixels is NULL (0), then we will allocate the buffer and return it to you.
538//--------------------------------
539
540STBIRDEF unsigned char * stbir_resize_uint8_srgb( const unsigned char *input_pixels , int input_w , int input_h, stbir_int64 input_stride_in_bytes,
541 unsigned char *output_pixels, int output_w, int output_h, stbir_int64 output_stride_in_bytes,
542 stbir_pixel_layout pixel_type );
543
544STBIRDEF unsigned char * stbir_resize_uint8_linear( const unsigned char *input_pixels , int input_w , int input_h, stbir_int64 input_stride_in_bytes,
545 unsigned char *output_pixels, int output_w, int output_h, stbir_int64 output_stride_in_bytes,
546 stbir_pixel_layout pixel_type );
547
548STBIRDEF float * stbir_resize_float_linear( const float *input_pixels , int input_w , int input_h, stbir_int64 input_stride_in_bytes,
549 float *output_pixels, int output_w, int output_h, stbir_int64 output_stride_in_bytes,
550 stbir_pixel_layout pixel_type );
551//===============================================================
552
553//===============================================================
554// Medium-complexity API
555//
556// This extends the easy-to-use API as follows:
557//
558// * Can specify the datatype - U8, U8_SRGB, U16, FLOAT, HALF_FLOAT
559// * Edge wrap can selected explicitly
560// * Filter can be selected explicitly
561//--------------------------------
562
563typedef enum
564{
565 STBIR_EDGE_CLAMP = 0,
566 STBIR_EDGE_REFLECT = 1,
567 STBIR_EDGE_WRAP = 2, // this edge mode is slower and uses more memory
568 STBIR_EDGE_ZERO = 3,
569} stbir_edge;
570
571typedef enum
572{
573 STBIR_FILTER_DEFAULT = 0, // use same filter type that easy-to-use API chooses
574 STBIR_FILTER_BOX = 1, // A trapezoid w/1-pixel wide ramps, same result as box for integer scale ratios
575 STBIR_FILTER_TRIANGLE = 2, // On upsampling, produces same results as bilinear texture filtering
576 STBIR_FILTER_CUBICBSPLINE = 3, // The cubic b-spline (aka Mitchell-Netrevalli with B=1,C=0), gaussian-esque
577 STBIR_FILTER_CATMULLROM = 4, // An interpolating cubic spline (the default when upsampling)
578 STBIR_FILTER_MITCHELL = 5, // Mitchell-Netrevalli filter with B=1/3, C=1/3 (the default when downsampling)
579 STBIR_FILTER_POINT_SAMPLE = 6, // Simple point sampling
580 STBIR_FILTER_MITCHELL_QUARTER = 7, // Mitchell-Netrevalli with B=1/4, C=3/8 (Robidoux), a little sharper than Mitchell
581 STBIR_FILTER_MITCHELL_SIXTH = 8, // Mitchell-Netrevalli with B=1/6, C=5/12, sharper still
582 STBIR_FILTER_MITCHELL_NEG_SIXTH = 9, // Mitchell-Netrevalli with B=-1/6, C=7/12, over-sharpening
583 STBIR_FILTER_MITCHELL_NEG_THIRD = 10, // Mitchell-Netrevalli with B=-1/3, C=2/3, more over-sharpening
584 STBIR_FILTER_LANCZOS4 = 11, // Lanczos with 2 lobes (4 taps at 1:1), sharp, some ringing on hard edges
585 STBIR_FILTER_LANCZOS5 = 12, // Lanczos with 2.5 lobes (5 taps at 1:1), very sharp, often ringy
586 STBIR_FILTER_MURATORI6 = 13, // Casey Muratori's Bink 2 kernel (1 -4 19 19 -4 1)/32: sharper than Catmull-Rom, stable under repeated half pixel shifts when using 8-bit pixels
587 STBIR_FILTER_OTHER = 14, // User callback specified (keep last - the built-ins are everything below it)
588} stbir_filter;
589
590typedef enum
591{
592 STBIR_TYPE_UINT8 = 0,
593 STBIR_TYPE_UINT8_SRGB = 1,
594 STBIR_TYPE_UINT8_SRGB_ALPHA = 2, // alpha channel, when present, should also be SRGB (this is very unusual)
595 STBIR_TYPE_UINT16 = 3,
596 STBIR_TYPE_FLOAT = 4,
597 STBIR_TYPE_HALF_FLOAT = 5
598} stbir_datatype;
599
600// medium api
601STBIRDEF void * stbir_resize( const void *input_pixels , int input_w , int input_h, stbir_int64 input_stride_in_bytes,
602 void *output_pixels, int output_w, int output_h, stbir_int64 output_stride_in_bytes,
603 stbir_pixel_layout pixel_layout, stbir_datatype data_type,
604 stbir_edge edge, stbir_filter filter );
605//===============================================================
606
607
608//===============================================================
609// Extended-complexity API
610//
611// This API exposes all resize functionality.
612//
613// * Separate filter types for each axis
614// * Separate edge modes for each axis
615// * Separate input and output data types
616// * Can specify regions with subpixel correctness
617// * Can specify alpha flags
618// * Can specify a memory callback
619// * Can specify a callback data type for pixel input and output
620// * Can be threaded for a single resize
621// * Can be used to resize many frames without recalculating the sampler info
622//
623// Use this API as follows:
624// 1) Call the stbir_resize_init function on a local STBIR_RESIZE structure
625// 2) Call any of the stbir_set functions
626// 3) Optionally call stbir_build_samplers() if you are going to resample multiple times
627// with the same input and output dimensions (like resizing video frames)
628// 4) Resample by calling stbir_resize_extended().
629// 5) Call stbir_free_samplers() if you called stbir_build_samplers()
630//--------------------------------
631
632
633// Types:
634
635// INPUT CALLBACK: this callback is used for input scanlines
636typedef void const * stbir_input_callback( void * optional_output, void const * input_ptr, int num_pixels, int x, int y, void * context );
637
638// OUTPUT CALLBACK: this callback is used for output scanlines (or pieces of scanlines - see below)
639typedef void stbir_output_callback( void const * output_ptr, int num_pixels, int x, int y, void * context );
640
641// callbacks for user installed filters (there are old and new versions, please use the new versions from now on)
642typedef float stbir__kernel_callback( float x, float scale, void * user_data );
643typedef float stbir__support_callback( float scale, void * user_data );
644typedef stbir_real stbir__kernel_callback_new( stbir_real x, stbir_real scale, void * user_data );
645typedef stbir_real stbir__support_callback_new( stbir_real scale, void * user_data );
646typedef stbir_real stbir__radius_callback_new( stbir_real scale, stbir_real inv_scale, int is_upsample, void * user_data );
647
648// internal structure with precomputed scaling
649typedef struct stbir__info stbir__info;
650
651typedef struct STBIR_RESIZE // use the stbir_resize_init and stbir_override functions to set these values for future compatibility
652{
653 void * user_data;
654 void const * input_pixels;
655 int input_w, input_h;
656 stbir_real input_s0, input_t0, input_s1, input_t1;
657 int input_s_n0, input_s_n1, input_s_d; // the input region as fractions, when it was set in pixels: s0 = n0/d, s1 = n1/d (d is 0 when set as reals)
658 int input_t_n0, input_t_n1, input_t_d;
659 stbir_input_callback * input_cb;
660 void * output_pixels;
661 int output_w, output_h;
662 int output_subx, output_suby, output_subw, output_subh;
663 stbir_output_callback * output_cb;
664 stbir_int64 input_stride_in_bytes;
665 stbir_int64 output_stride_in_bytes;
666 int splits;
667 int fast_alpha;
668 int needs_rebuild;
669 int called_alloc;
670 stbir_pixel_layout input_pixel_layout_public;
671 stbir_pixel_layout output_pixel_layout_public;
672 stbir_datatype input_data_type;
673 stbir_datatype output_data_type;
674 stbir_filter horizontal_filter, vertical_filter;
675 stbir_edge horizontal_edge, vertical_edge;
676 stbir__kernel_callback * horizontal_filter_kernel; stbir__support_callback * horizontal_filter_support; // old callback form
677 stbir__kernel_callback * vertical_filter_kernel; stbir__support_callback * vertical_filter_support;
678 stbir__kernel_callback_new * horizontal_filter_kernel_new; stbir__support_callback_new * horizontal_filter_support_new; stbir__radius_callback_new * horizontal_filter_radius;
679 stbir__kernel_callback_new * vertical_filter_kernel_new; stbir__support_callback_new * vertical_filter_support_new; stbir__radius_callback_new * vertical_filter_radius;
680 stbir__info * samplers;
681} STBIR_RESIZE;
682
683// extended complexity api
684
685
686// First off, you must ALWAYS call stbir_resize_init on your resize structure before any of the other calls!
687STBIRDEF void stbir_resize_init( STBIR_RESIZE * resize,
688 const void *input_pixels, int input_w, int input_h, stbir_int64 input_stride_in_bytes, // stride can be zero
689 void *output_pixels, int output_w, int output_h, stbir_int64 output_stride_in_bytes, // stride can be zero
690 stbir_pixel_layout pixel_layout, stbir_datatype data_type );
691
692//===============================================================
693// You can update these parameters any time after resize_init and there is no cost
694//--------------------------------
695
696STBIRDEF void stbir_set_datatypes( STBIR_RESIZE * resize, stbir_datatype input_type, stbir_datatype output_type );
697STBIRDEF void stbir_set_pixel_callbacks( STBIR_RESIZE * resize, stbir_input_callback * input_cb, stbir_output_callback * output_cb ); // no callbacks by default
698STBIRDEF void stbir_set_user_data( STBIR_RESIZE * resize, void * user_data ); // pass back STBIR_RESIZE* by default
699STBIRDEF void stbir_set_buffer_ptrs( STBIR_RESIZE * resize, const void * input_pixels, stbir_int64 input_stride_in_bytes, void * output_pixels, stbir_int64 output_stride_in_bytes );
700
701//===============================================================
702
703
704//===============================================================
705// If you call any of these functions, you will trigger a sampler rebuild!
706//--------------------------------
707
708STBIRDEF int stbir_set_pixel_layouts( STBIR_RESIZE * resize, stbir_pixel_layout input_pixel_layout, stbir_pixel_layout output_pixel_layout ); // sets new buffer layouts
709STBIRDEF int stbir_set_edgemodes( STBIR_RESIZE * resize, stbir_edge horizontal_edge, stbir_edge vertical_edge ); // CLAMP by default
710
711STBIRDEF int stbir_set_filters( STBIR_RESIZE * resize, stbir_filter horizontal_filter, stbir_filter vertical_filter ); // STBIR_DEFAULT_FILTER_UPSAMPLE/DOWNSAMPLE by default
712
713// Your own filters: per axis, a kernel callback and one of two ways to give its extent (a support or a radius - both
714// on one axis returns 0; all null keeps the built-in filter). All in stbir_real (double unless user overridden to float).
715// the resizer places the taps in stbir_real and rounds each weight to float once, into the coefficient table.
716//
717// The kernel returns the weight at x, in the filter's own pixels (output pixels when downsampling, input pixels when
718// upsampling), centered on zero. It is also passed the ratio <= 1 (output / input downsampling, input / output
719// upsampling), for kernels like the box whose shape depends on it.
720//
721// The support returns the filter's radius in those same filter pixels, and is passed the same ratio. For most kernels
722// this is a constant (2 for the cubics, 1 for the triangle); the resizer converts it to input pixels itself.
723//
724// The radius is the alternative for a kernel whose extent depends on the scale: it returns the radius in INPUT pixels,
725// from scale, inv_scale and the resizer's own up / down decision (is_upsample, 1:1 included), so an exact ratio gets
726// an exact tap count (the built-in box is 0.5 * inv_scale + 0.5).
727//
728// static stbir_real my_kernel( stbir_real x, stbir_real scale, void * user ) { ... }
729// static stbir_real my_support( stbir_real scale, void * user ) { return 2.0; }
730// stbir_set_filter_callbacks_new( &resize, my_kernel, my_support, 0, my_kernel, my_support, 0 );
731STBIRDEF int stbir_set_filter_callbacks_new( STBIR_RESIZE * resize, stbir__kernel_callback_new * horizontal_filter, stbir__support_callback_new * horizontal_support, stbir__radius_callback_new * horizontal_radius, stbir__kernel_callback_new * vertical_filter, stbir__support_callback_new * vertical_support, stbir__radius_callback_new * vertical_radius );
732
733// Old version of the filter callbacks (for backwards compatibility) - please use stbir_set_filter_callbacks_new for new code.
734STBIRDEF int stbir_set_filter_callbacks( STBIR_RESIZE * resize, stbir__kernel_callback * horizontal_filter, stbir__support_callback * horizontal_support, stbir__kernel_callback * vertical_filter, stbir__support_callback * vertical_support );
735
736STBIRDEF int stbir_set_pixel_subrect( STBIR_RESIZE * resize, int subx, int suby, int subw, int subh ); // sets both sub-regions (full regions by default)
737STBIRDEF int stbir_set_input_subrect( STBIR_RESIZE * resize, stbir_real s0, stbir_real t0, stbir_real s1, stbir_real t1 ); // sets input sub-region (full region by default). a region past the image is clipped to it: the output pixels mapping outside aren't written
738STBIRDEF int stbir_set_output_pixel_subrect( STBIR_RESIZE * resize, int subx, int suby, int subw, int subh ); // sets output sub-region (full region by default)
739
740// when inputting AND outputting non-premultiplied alpha pixels, we use a slower but higher quality technique
741// that fills the zero alpha pixel's RGB values with something plausible. If you don't care about areas of
742// zero alpha, you can call this function to get about a 25% speed improvement for STBIR_RGBA to STBIR_RGBA
743// types of resizes.
744STBIRDEF int stbir_set_non_pm_alpha_speed_over_quality( STBIR_RESIZE * resize, int non_pma_alpha_speed_over_quality );
745//===============================================================
746
747
748//===============================================================
749// You can call build_samplers to prebuild all the internal data we need to resample.
750// Then, if you call resize_extended many times with the same resize, you only pay the
751// cost once.
752// If you do call build_samplers, you MUST call free_samplers eventually.
753//--------------------------------
754
755// This builds the samplers and does one allocation
756STBIRDEF int stbir_build_samplers( STBIR_RESIZE * resize );
757
758// You MUST call this, if you call stbir_build_samplers or stbir_build_samplers_with_splits
759STBIRDEF void stbir_free_samplers( STBIR_RESIZE * resize );
760//===============================================================
761
762
763// And this is the main function to perform the resize synchronously on one thread.
764STBIRDEF int stbir_resize_extended( STBIR_RESIZE * resize );
765
766
767//===============================================================
768// Use these functions for multithreading.
769// 1) You call stbir_build_samplers_with_splits first on the main thread
770// 2) Then stbir_resize_extended_split on each thread
771// 3) stbir_free_samplers when done on the main thread
772//--------------------------------
773
774// This will build samplers for threading.
775// You can pass in the number of threads you'd like to use (try_splits).
776// It returns the number of splits (threads) that you can call it with.
777// It might be less: when the image can't be split up that many ways, or when
778// the build's estimate says more threads wouldn't help.
779
780STBIRDEF int stbir_build_samplers_with_splits( STBIR_RESIZE * resize, int try_splits );
781
782// This function does a split of the resizing (you call this fuction for each
783// split, on multiple threads). A split is a piece of the output resize pixel space.
784
785// Note that you MUST call stbir_build_samplers_with_splits before stbir_resize_extended_split!
786
787// Usually, you will always call stbir_resize_extended_split with split_start as the thread_index
788// and "1" for the split_count.
789// But, if you have a weird situation where you MIGHT want 8 threads, but sometimes
790// only 4 threads, you can use 0,2,4,6 for the split_start's and use "2" for the
791// split_count each time to turn in into a 4 thread resize. (This is unusual).
792
793STBIRDEF int stbir_resize_extended_split( STBIR_RESIZE * resize, int split_start, int split_count );
794//===============================================================
795
796
797//===============================================================
798// Pixel Callbacks info:
799//--------------------------------
800
801// The input callback is super flexible - it calls you with the input address
802// of the start of the input row (based on the stride and base pointer), and the
803// x position and pixel count that it needs from that row. It gives you an
804// optional_output pointer that you can fill, or you can just return your own
805// pointer into your own data.
806//
807// You can also do conversion from non-supported data types if necessary - in
808// this case, you ignore the input_ptr and just use the x and y parameters to
809// calculate your own input_ptr based on the size of each non-supported pixel.
810// (Something like the third example below.)
811//
812// You can also install just an input or just an output callback by setting the
813// callback that you don't want to zero.
814//
815// First example, progress: (getting a callback that you can monitor the progress):
816// void const * my_callback( void * optional_output, void const * input_ptr, int num_pixels, int x, int y, void * context )
817// {
818// percentage_done = y / input_height;
819// return ( (char*) input_ptr ) + x * pixel_width_in_bytes; // use buffer from call (input_ptr is the row start)
820// }
821//
822// Next example, copying: (copy from some other buffer or stream):
823// void const * my_callback( void * optional_output, void const * input_ptr, int num_pixels, int x, int y, void * context )
824// {
825// CopyOrStreamData( optional_output, other_data_src, num_pixels * pixel_width_in_bytes );
826// return optional_output; // return the optional buffer that we filled
827// }
828//
829// Third example, input another buffer without copying: (zero-copy from other buffer):
830// void const * my_callback( void * optional_output, void const * input_ptr, int num_pixels, int x, int y, void * context )
831// {
832// void * pixels = ( (char*) other_image_base ) + ( y * other_image_stride ) + ( x * other_pixel_width_in_bytes );
833// return pixels; // return pointer to your data without copying
834// }
835//
836//
837// The output callback is considerably simpler - it just calls you so that you can dump
838// out the output pixels. You can also convert to other output types here if you want.
839//
840// Note that the resizer can process the output in column slices (to stay in cache), so
841// you are called with pieces of scanlines: num_pixels pixels starting at output pixel x
842// on output row y. x and y are always in the full output image's pixel space (so with an
843// output sub-rect, they include the sub-rect offset). The pieces of a scanline are not
844// delivered in order, so if you want whole scanlines, assemble them yourself with x and y.
845//
846// Simple example:
847// void my_output( void const * output_ptr, int num_pixels, int x, int y, void * context )
848// {
849// memcpy( ( (char*) my_output_image ) + y * my_output_stride + x * pixel_width_in_bytes, output_ptr, num_pixels * pixel_width_in_bytes );
850// }
851//===============================================================
852
853
854
855//===============================================================
856// optional built-in profiling API
857//--------------------------------
858
859#ifdef STBIR_PROFILE
860
861typedef struct STBIR_PROFILE_INFO
862{
863 stbir_uint64 total_clocks;
864
865 // how many clocks spent (of total_clocks) in the various resize routines, along with a string description
866 // there are "resize_count" number of zones
867 stbir_uint64 clocks[ 8 ];
868 char const ** descriptions;
869
870 // count of clocks and descriptions
871 stbir_uint32 count;
872} STBIR_PROFILE_INFO;
873
874// use after calling stbir_resize_extended (or stbir_build_samplers or stbir_build_samplers_with_splits)
875STBIRDEF void stbir_resize_build_profile_info( STBIR_PROFILE_INFO * out_info, STBIR_RESIZE const * resize );
876
877// use after calling stbir_resize_extended
878STBIRDEF void stbir_resize_extended_profile_info( STBIR_PROFILE_INFO * out_info, STBIR_RESIZE const * resize );
879
880// use after calling stbir_resize_extended_split
881STBIRDEF void stbir_resize_split_profile_info( STBIR_PROFILE_INFO * out_info, STBIR_RESIZE const * resize, int split_start, int split_num );
882
883//===============================================================
884
885#endif
886
887
888//// end header file /////////////////////////////////////////////////////
889#endif // STBIR_INCLUDE_STB_IMAGE_RESIZE2_H
890
891#if defined(STB_IMAGE_RESIZE_IMPLEMENTATION) || defined(STB_IMAGE_RESIZE2_IMPLEMENTATION)
892
893#ifndef STBIR_ASSERT
894#include <assert.h>
895#define STBIR_ASSERT(x) assert(x)
896#endif
897
898#ifndef STBIR_MALLOC
899#include <stdlib.h>
900#define STBIR_MALLOC(size,user_data) ((void)(user_data), malloc(size))
901#define STBIR_FREE(ptr,user_data) ((void)(user_data), free(ptr))
902// (we used the comma operator to evaluate user_data, to avoid "unused parameter" warnings)
903#endif
904
905#ifdef _MSC_VER
906
907#define stbir__inline __forceinline
908
909#else
910
911#define stbir__inline __inline__
912
913// Clang address sanitizer
914#if defined(__has_feature)
915 #if __has_feature(address_sanitizer) || __has_feature(memory_sanitizer)
916 #ifndef STBIR__SEPARATE_ALLOCATIONS
917 #define STBIR__SEPARATE_ALLOCATIONS
918 #endif
919 #endif
920#endif
921
922#endif
923
924// GCC and MSVC
925#if defined(__SANITIZE_ADDRESS__)
926 #ifndef STBIR__SEPARATE_ALLOCATIONS
927 #define STBIR__SEPARATE_ALLOCATIONS
928 #endif
929#endif
930
931// Always turn off automatic FMA use - use STBIR_USE_FMA if you want.
932// Otherwise, this is a determinism disaster.
933#ifndef STBIR_DONT_CHANGE_FP_CONTRACT // override in case you don't want this behavior
934#if defined(_MSC_VER) && !defined(__clang__)
935#if _MSC_VER > 1200
936#pragma fp_contract(off)
937#endif
938#elif defined(__GNUC__) && !defined(__clang__)
939#pragma GCC optimize("fp-contract=off")
940#else
941#pragma STDC FP_CONTRACT OFF
942#endif
943#endif
944
945#ifdef _MSC_VER
946#define STBIR__UNUSED(v) (void)(v)
947#else
948#define STBIR__UNUSED(v) (void)sizeof(v)
949#endif
950
951#define STBIR__ARRAY_SIZE(a) (sizeof((a))/sizeof((a)[0]))
952
953
954#ifndef STBIR_DEFAULT_FILTER_UPSAMPLE
955#define STBIR_DEFAULT_FILTER_UPSAMPLE STBIR_FILTER_CATMULLROM
956#endif
957
958#ifndef STBIR_DEFAULT_FILTER_DOWNSAMPLE
959#define STBIR_DEFAULT_FILTER_DOWNSAMPLE STBIR_FILTER_MITCHELL
960#endif
961
962
963#ifndef STBIR__HEADER_FILENAME
964#define STBIR__HEADER_FILENAME "stb_image_resize2.h"
965#endif
966
967// the internal pixel layout enums are in a different order, so we can easily do range comparisons of types
968// the public pixel layout is ordered in a way that if you cast num_channels (1-4) to the enum, you get something sensible
969typedef enum
970{
971 STBIRI_1CHANNEL = 0,
972 STBIRI_2CHANNEL = 1,
973 STBIRI_RGB = 2,
974 STBIRI_BGR = 3,
975 STBIRI_4CHANNEL = 4,
976
977 STBIRI_RGBA = 5,
978 STBIRI_BGRA = 6,
979 STBIRI_ARGB = 7,
980 STBIRI_ABGR = 8,
981 STBIRI_RA = 9,
982 STBIRI_AR = 10,
983
984 STBIRI_RGBA_PM = 11,
985 STBIRI_BGRA_PM = 12,
986 STBIRI_ARGB_PM = 13,
987 STBIRI_ABGR_PM = 14,
988 STBIRI_RA_PM = 15,
989 STBIRI_AR_PM = 16,
990} stbir_internal_pixel_layout;
991
992// define the public pixel layouts to not compile inside the implementation (to avoid accidental use)
993#define STBIR_BGR bad_dont_use_in_implementation
994#define STBIR_1CHANNEL STBIR_BGR
995#define STBIR_2CHANNEL STBIR_BGR
996#define STBIR_RGB STBIR_BGR
997#define STBIR_RGBA STBIR_BGR
998#define STBIR_4CHANNEL STBIR_BGR
999#define STBIR_BGRA STBIR_BGR
1000#define STBIR_ARGB STBIR_BGR
1001#define STBIR_ABGR STBIR_BGR
1002#define STBIR_RA STBIR_BGR
1003#define STBIR_AR STBIR_BGR
1004#define STBIR_RGBA_PM STBIR_BGR
1005#define STBIR_BGRA_PM STBIR_BGR
1006#define STBIR_ARGB_PM STBIR_BGR
1007#define STBIR_ABGR_PM STBIR_BGR
1008#define STBIR_RA_PM STBIR_BGR
1009#define STBIR_AR_PM STBIR_BGR
1010
1011// must match stbir_datatype
1012static unsigned char stbir__type_size[] = {
1013 1,1,1,2,4,2 // STBIR_TYPE_UINT8,STBIR_TYPE_UINT8_SRGB,STBIR_TYPE_UINT8_SRGB_ALPHA,STBIR_TYPE_UINT16,STBIR_TYPE_FLOAT,STBIR_TYPE_HALF_FLOAT
1014};
1015
1016// When gathering, the contributors are which source pixels contribute.
1017// When scattering, the contributors are which destination pixels are contributed to.
1018typedef struct
1019{
1020 int n0; // First contributing pixel
1021 int n1; // Last contributing pixel
1022} stbir__contributors;
1023
1024typedef struct
1025{
1026 int lowest; // First sample index for whole filter
1027 int highest; // Last sample index for whole filter
1028 int widest; // widest single set of samples for an output
1029} stbir__filter_extent_info;
1030
1031typedef struct stbir__scale_info
1032{
1033 int input_full_size;
1034 int output_sub_size;
1035 stbir_real scale;
1036 stbir_real inv_scale;
1037 stbir_real pixel_shift; // starting shift in output pixel space (in pixels)
1038 int scale_is_rational;
1039 stbir_uint32 scale_numerator, scale_denominator;
1040 stbir_real tap_scale_in, tap_scale_out; // 1 / 2N and 1 / 2D for stbir__tap_offset
1041 int positions_exact; // the tap positions come from the exact fraction and an integer shift (stbir__tap_offset)
1042 int shift_pixels; // pixel_shift as that integer
1043 int approximated_drift; // the fraction approximates the scale: how far its far edge can land from the float chain's, in input pixels (0: exact)
1044} stbir__scale_info;
1045
1046typedef struct
1047{
1048 stbir__contributors * contributors;
1049 float* coefficients;
1050 stbir__scale_info scale_info;
1051 stbir_filter filter_enum;
1052 stbir__kernel_callback_new * filter_kernel_new; // the kernel: the built-in's or the user's, or
1053 stbir__radius_callback_new * filter_radius; // the filter's extent, one of the three: a radius in input pixels (the box) or
1054 stbir__support_callback_new * filter_support_new; // a support in filter pixels (the other built-ins); the user's, or the built-in's,
1055 stbir__kernel_callback * filter_kernel_old; // an old back compat float kernel
1056 stbir__support_callback * filter_support_old; // or an old back compat support function
1057 stbir_edge edge;
1058 int coefficient_width;
1059 int entry_floats; // horizontal only: floats per packed column entry (STBIR__COLUMN_ENTRY_FLOATS of coefficient_width)
1060
1061 // polyphase sharing (see stbir__plan_polyphase_sharing). The coefficient sets for the repeating part of a rational scale
1062 // are stored once. 0 = every output has its own set.
1063 int shared_period; // coefficient sets in the repeating block
1064 int shared_prologue; // outputs with their own coefficient sets before it
1065 int shared_epilogue; // outputs with their own coefficient sets after it
1066 int shared_block_n0; // n0 of the block's first output (output P), and how far n0 moves per pass over the block
1067 int shared_block_shift;
1068 int shared_epilogue_n0; // n0 of the first epilogue output
1069 int filter_pixel_width;
1070 int filter_pixel_margin;
1071 int num_contributors;
1072 size_t contributors_size;
1073 size_t coefficients_size;
1074 stbir__filter_extent_info extent_info;
1075 int is_gather; // 0 = scatter, 1 = gather with scale >= 1, 2 = gather with scale < 1
1076} stbir__sampler;
1077
1078// horizontal routines combine the contribs and coeffs into one structure - coefficients is actually an array of width floats.
1079typedef struct
1080{
1081 int advance; // how far the running decode pointer moves for the NEXT column, in floats (already times the channel count)
1082 int width; // how many pixels this column reads (n1 - n0 + 1) - only the n-coefficient kernels look at it
1083 float coefficients[ 1 ]; // width floats long, with a zero pad after for the widths whose kernels load one float past
1084} stbir__column_entry;
1085
1086// a column slice is a run of output columns that the whole vertical/horizontal loop is run over at once (instead of
1087// full scanlines), so the ring buffer and the temp buffers stay in cache. the coefficients are shared by all the
1088// slices, each slice just starts at a different offset into them.
1089typedef struct
1090{
1091 int x0; // first output pixel of the slice
1092 int width; // output pixels in the slice
1093 int in_n0, in_n1; // input pixels decoded for the slice (in the horizontal sampler's coordinate space - can hang off either edge with WRAP)
1094 int first_n0; // input pixel output x0 starts reading at
1095} stbir__column_slice;
1096
1097typedef struct
1098{
1099#ifdef STBIR_PROFILE
1100 union
1101 {
1102 struct { stbir_uint64 total, looping, vertical, horizontal, decode, encode, alpha, unalpha; } named;
1103 stbir_uint64 array[8];
1104 } profile;
1105 stbir_uint64 * current_zone_excluded_ptr;
1106#endif
1107 float* decode_buffer;
1108
1109 int ring_buffer_first_scanline;
1110 int ring_buffer_last_scanline;
1111 int ring_buffer_begin_index; // first_scanline is at this index in the ring buffer
1112 int start_output_y, end_output_y;
1113 int start_input_y, end_input_y; // used in scatter only
1114 int first_column_slice, num_column_slices; // the column slices this split runs: all of them when the splits are bands of rows, its own range when they are columns
1115
1116 #ifdef STBIR__SEPARATE_ALLOCATIONS
1117 float** ring_buffers; // one pointer for each ring buffer
1118 #else
1119 float* ring_buffer; // one big buffer that we index into
1120 #endif
1121
1122 float* vertical_buffer;
1123
1124 // where the buffers above were allocated. they each have a page of slack, and get repositioned within it at
1125 // resize time to keep them clear of 4K aliasing with the input rows (see stbir__4K_alias_skew_buffers)
1126 void* decode_buffer_alloc;
1127 void* ring_buffer_alloc;
1128 void* vertical_buffer_alloc;
1129
1130 char no_cache_straddle[64];
1131} stbir__per_split_info;
1132
1133typedef float * stbir__decode_pixels_func( float * decode, ptrdiff_t width_times_channels, void const * input );
1134typedef void stbir__alpha_weight_func( float * decode_buffer, ptrdiff_t width_times_channels );
1135typedef void stbir__horizontal_gather_channels_func( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer,
1136 stbir__column_entry const * column, int entry_stride );
1137typedef void stbir__alpha_unweight_func(float * encode_buffer, ptrdiff_t width_times_channels );
1138typedef void stbir__encode_pixels_func( void * output, ptrdiff_t width_times_channels, float const * encode );
1139
1140struct stbir__info
1141{
1142#ifdef STBIR_PROFILE
1143 union
1144 {
1145 struct { stbir_uint64 total, build, alloc, horizontal, vertical, cleanup, scatter; } named;
1146 stbir_uint64 array[7];
1147 } profile;
1148 stbir_uint64 * current_zone_excluded_ptr;
1149#endif
1150 stbir__sampler horizontal;
1151 stbir__sampler vertical;
1152
1153 void const * input_data;
1154 void * output_data;
1155
1156 stbir_int64 input_stride_bytes;
1157 stbir_int64 output_stride_bytes;
1158 size_t ring_buffer_length_bytes; // The length of an individual entry in the ring buffer (there are ring_buffer_num_entries of them)
1159 int ring_buffer_num_entries; // Total number of entries in the ring buffer.
1160
1161 stbir_datatype input_type;
1162 stbir_datatype output_type;
1163
1164 stbir_input_callback * in_pixels_cb;
1165 void * user_data;
1166 stbir_output_callback * out_pixels_cb;
1167
1168 stbir__contributors conservative_extents; // input pixel range that covers everything the horizontal filter can touch (decode buffers are sized from this)
1169
1170 void * alloced_mem;
1171 stbir__per_split_info * split_info; // by default 1, but there will be N of these allocated based on the thread init you did
1172 stbir__column_slice * column_slices; // the output columns, left to right (the last slice can be narrower)
1173 int num_column_slices;
1174 int split_cols, split_rows; // the split grid: splits = split_cols column groups x split_rows row bands (see stbir__split_mode)
1175
1176 stbir__decode_pixels_func * decode_pixels;
1177 stbir__alpha_weight_func * alpha_weight;
1178 stbir__horizontal_gather_channels_func * horizontal_gather_channels;
1179 stbir__horizontal_gather_channels_func * horizontal_gather_period1; // for the run over a polyphase period 1 set (integer downsamples: one coefficient set for every output), or null
1180 stbir__alpha_unweight_func * alpha_unweight;
1181 stbir__encode_pixels_func * encode_pixels;
1182
1183 int alloc_ring_buffer_num_entries; // Number of entries in the ring buffer that will be allocated
1184 int splits; // count of splits
1185
1186 stbir_internal_pixel_layout input_pixel_layout_internal;
1187 stbir_internal_pixel_layout output_pixel_layout_internal;
1188
1189 int input_color_and_type;
1190 int input_callback_offset_bytes; // where the input callback's temp buffer sits relative to the decode target (see STBIR_INPUT_CALLBACK_PADDING)
1191 int offset_x, offset_y; // offset within output_data
1192 int vertical_first;
1193 int channels;
1194 int effective_channels; // same as channels, except on RGBA/ARGB (7), or XA/AX (3)
1195 size_t alloced_total;
1196};
1197
1198
1199#define stbir__max_uint8_as_float 255.0f
1200#define stbir__max_uint16_as_float 65535.0f
1201#define stbir__max_uint8_as_float_inverted 3.9215689e-03f // (1.0f/255.0f)
1202#define stbir__max_uint16_as_float_inverted 1.5259022e-05f // (1.0f/65535.0f)
1203
1204// an alpha below this isn't divided by when un-premultiplying - alpha is effectively zero
1205#define stbir__alpha_too_small_to_divide ((float)1 / (1 << 20) / (1 << 20) / (1 << 20) / (1 << 20) / (1 << 20) / (1 << 20))
1206
1207// a coefficient set whose sum is within this of 1 isn't renormalized: scaling by 1 / sum would move every float
1208// coefficient by less than half an ulp ( 2^-25 of itself ), so each would round straight back - the loop is a no-op
1209#define stbir__renorm_tolerance ( (stbir_real)1.0 / ( 1 << 26 ) )
1210
1211// a kernel value this small (either sign) is dropped before the coefficients are normalized
1212#define stbir__too_small_coefficient ( (stbir_real)1.0 / ( 1 << 26 ) )
1213
1214// a coefficient set whose sum is under this (either sign) is zeroed instead of normalized
1215#define stbir__too_small_coefficient_sum ( (stbir_real)1.0 / ( 1 << 16 ) )
1216
1217// min/max friendly
1218#define STBIR_CLAMP(x, xmin, xmax) for(;;) { \
1219 if ( (x) < (xmin) ) (x) = (xmin); \
1220 if ( (x) > (xmax) ) (x) = (xmax); \
1221 break; \
1222}
1223
1224static stbir__inline int stbir__min(int a, int b)
1225{
1226 return a < b ? a : b;
1227}
1228
1229static stbir__inline int stbir__max(int a, int b)
1230{
1231 return a > b ? a : b;
1232}
1233
1234static float stbir__srgb_uchar_to_linear_float[256] = {
1235 0.000000f, 0.000304f, 0.000607f, 0.000911f, 0.001214f, 0.001518f, 0.001821f, 0.002125f, 0.002428f, 0.002732f, 0.003035f,
1236 0.003347f, 0.003677f, 0.004025f, 0.004391f, 0.004777f, 0.005182f, 0.005605f, 0.006049f, 0.006512f, 0.006995f, 0.007499f,
1237 0.008023f, 0.008568f, 0.009134f, 0.009721f, 0.010330f, 0.010960f, 0.011612f, 0.012286f, 0.012983f, 0.013702f, 0.014444f,
1238 0.015209f, 0.015996f, 0.016807f, 0.017642f, 0.018500f, 0.019382f, 0.020289f, 0.021219f, 0.022174f, 0.023153f, 0.024158f,
1239 0.025187f, 0.026241f, 0.027321f, 0.028426f, 0.029557f, 0.030713f, 0.031896f, 0.033105f, 0.034340f, 0.035601f, 0.036889f,
1240 0.038204f, 0.039546f, 0.040915f, 0.042311f, 0.043735f, 0.045186f, 0.046665f, 0.048172f, 0.049707f, 0.051269f, 0.052861f,
1241 0.054480f, 0.056128f, 0.057805f, 0.059511f, 0.061246f, 0.063010f, 0.064803f, 0.066626f, 0.068478f, 0.070360f, 0.072272f,
1242 0.074214f, 0.076185f, 0.078187f, 0.080220f, 0.082283f, 0.084376f, 0.086500f, 0.088656f, 0.090842f, 0.093059f, 0.095307f,
1243 0.097587f, 0.099899f, 0.102242f, 0.104616f, 0.107023f, 0.109462f, 0.111932f, 0.114435f, 0.116971f, 0.119538f, 0.122139f,
1244 0.124772f, 0.127438f, 0.130136f, 0.132868f, 0.135633f, 0.138432f, 0.141263f, 0.144128f, 0.147027f, 0.149960f, 0.152926f,
1245 0.155926f, 0.158961f, 0.162029f, 0.165132f, 0.168269f, 0.171441f, 0.174647f, 0.177888f, 0.181164f, 0.184475f, 0.187821f,
1246 0.191202f, 0.194618f, 0.198069f, 0.201556f, 0.205079f, 0.208637f, 0.212231f, 0.215861f, 0.219526f, 0.223228f, 0.226966f,
1247 0.230740f, 0.234551f, 0.238398f, 0.242281f, 0.246201f, 0.250158f, 0.254152f, 0.258183f, 0.262251f, 0.266356f, 0.270498f,
1248 0.274677f, 0.278894f, 0.283149f, 0.287441f, 0.291771f, 0.296138f, 0.300544f, 0.304987f, 0.309469f, 0.313989f, 0.318547f,
1249 0.323143f, 0.327778f, 0.332452f, 0.337164f, 0.341914f, 0.346704f, 0.351533f, 0.356400f, 0.361307f, 0.366253f, 0.371238f,
1250 0.376262f, 0.381326f, 0.386430f, 0.391573f, 0.396755f, 0.401978f, 0.407240f, 0.412543f, 0.417885f, 0.423268f, 0.428691f,
1251 0.434154f, 0.439657f, 0.445201f, 0.450786f, 0.456411f, 0.462077f, 0.467784f, 0.473532f, 0.479320f, 0.485150f, 0.491021f,
1252 0.496933f, 0.502887f, 0.508881f, 0.514918f, 0.520996f, 0.527115f, 0.533276f, 0.539480f, 0.545725f, 0.552011f, 0.558340f,
1253 0.564712f, 0.571125f, 0.577581f, 0.584078f, 0.590619f, 0.597202f, 0.603827f, 0.610496f, 0.617207f, 0.623960f, 0.630757f,
1254 0.637597f, 0.644480f, 0.651406f, 0.658375f, 0.665387f, 0.672443f, 0.679543f, 0.686685f, 0.693872f, 0.701102f, 0.708376f,
1255 0.715694f, 0.723055f, 0.730461f, 0.737911f, 0.745404f, 0.752942f, 0.760525f, 0.768151f, 0.775822f, 0.783538f, 0.791298f,
1256 0.799103f, 0.806952f, 0.814847f, 0.822786f, 0.830770f, 0.838799f, 0.846873f, 0.854993f, 0.863157f, 0.871367f, 0.879622f,
1257 0.887923f, 0.896269f, 0.904661f, 0.913099f, 0.921582f, 0.930111f, 0.938686f, 0.947307f, 0.955974f, 0.964686f, 0.973445f,
1258 0.982251f, 0.991102f, 1.0f
1259};
1260
1261typedef union
1262{
1263 unsigned int u;
1264 float f;
1265} stbir__FP32;
1266
1267// From https://gist.github.com/rygorous/2203834
1268
1269// the SIMD sRGB encoders index this with a float's bits munged into 912..1015, so this biases it back to the table size
1270#define STBIR__SRGB8_TAB4_BIAS ((127-13)*8)
1271#define STBIR__SRGB8_TAB4( index ) fp32_to_srgb8_tab4[ (ptrdiff_t)(index) - STBIR__SRGB8_TAB4_BIAS ]
1272static const stbir_uint32 fp32_to_srgb8_tab4[104] = {
1273 0x0073000d, 0x007a000d, 0x0080000d, 0x0087000d, 0x008d000d, 0x0094000d, 0x009a000d, 0x00a1000d,
1274 0x00a7001a, 0x00b4001a, 0x00c1001a, 0x00ce001a, 0x00da001a, 0x00e7001a, 0x00f4001a, 0x0101001a,
1275 0x010e0033, 0x01280033, 0x01410033, 0x015b0033, 0x01750033, 0x018f0033, 0x01a80033, 0x01c20033,
1276 0x01dc0067, 0x020f0067, 0x02430067, 0x02760067, 0x02aa0067, 0x02dd0067, 0x03110067, 0x03440067,
1277 0x037800ce, 0x03df00ce, 0x044600ce, 0x04ad00ce, 0x051400ce, 0x057b00c5, 0x05dd00bc, 0x063b00b5,
1278 0x06970158, 0x07420142, 0x07e30130, 0x087b0120, 0x090b0112, 0x09940106, 0x0a1700fc, 0x0a9500f2,
1279 0x0b0f01cb, 0x0bf401ae, 0x0ccb0195, 0x0d950180, 0x0e56016e, 0x0f0d015e, 0x0fbc0150, 0x10630143,
1280 0x11070264, 0x1238023e, 0x1357021d, 0x14660201, 0x156601e9, 0x165a01d3, 0x174401c0, 0x182401af,
1281 0x18fe0331, 0x1a9602fe, 0x1c1502d2, 0x1d7e02ad, 0x1ed4028d, 0x201a0270, 0x21520256, 0x227d0240,
1282 0x239f0443, 0x25c003fe, 0x27bf03c4, 0x29a10392, 0x2b6a0367, 0x2d1d0341, 0x2ebe031f, 0x304d0300,
1283 0x31d105b0, 0x34a80555, 0x37520507, 0x39d504c5, 0x3c37048b, 0x3e7c0458, 0x40a8042a, 0x42bd0401,
1284 0x44c20798, 0x488e071e, 0x4c1c06b6, 0x4f76065d, 0x52a50610, 0x55ac05cc, 0x5892058f, 0x5b590559,
1285 0x5e0c0a23, 0x631c0980, 0x67db08f6, 0x6c55087f, 0x70940818, 0x74a007bd, 0x787d076c, 0x7c330723,
1286};
1287
1288static stbir__inline stbir_uint8 stbir__linear_to_srgb_uchar(float in)
1289{
1290 static const stbir__FP32 almostone = { 0x3f7fffff }; // 1-eps
1291 static const stbir__FP32 minval = { (127-13) << 23 };
1292 stbir_uint32 tab,bias,scale,t;
1293 stbir__FP32 f;
1294
1295 // Clamp to [2^(-13), 1-eps]; these two values map to 0 and 1, respectively.
1296 // The tests are carefully written so that NaNs map to 0, same as in the reference
1297 // implementation.
1298 if (!(in > minval.f)) // written this way to catch NaNs
1299 return 0;
1300 if (in > almostone.f)
1301 return 255;
1302
1303 // Do the table lookup and unpack bias, scale
1304 f.f = in;
1305 tab = fp32_to_srgb8_tab4[(f.u - minval.u) >> 20];
1306 bias = (tab >> 16) << 9;
1307 scale = tab & 0xffff;
1308
1309 // Grab next-highest mantissa bits and perform linear interpolation
1310 t = (f.u >> 12) & 0xff;
1311 return (unsigned char) ((bias + scale*t) >> 16);
1312}
1313
1314#ifndef STBIR_FORCE_GATHER_FILTER_SCANLINES_AMOUNT
1315#define STBIR_FORCE_GATHER_FILTER_SCANLINES_AMOUNT 128 // when downsampling and <= 128 scanlines of buffering, use gather (scatter above that - 32 or more to 1 downsampling)
1316#endif
1317
1318
1319// every scanline buffer carries this many extra floats past the end. we right justify into the buffer
1320// so the conversion can read ahead of the writes. also gives the horizontal kernels a one-float overread
1321#define STBIR_INPUT_CALLBACK_PADDING 16
1322
1323#ifdef _M_IX86_FP
1324#if ( _M_IX86_FP >= 2 )
1325#ifndef STBIR_SSE
1326#define STBIR_SSE
1327#endif
1328#endif
1329#endif
1330
1331#ifdef __TINYC__
1332 // tiny c has no intrinsics yet - this can become a version check if they add them
1333 #define STBIR_NO_SIMD
1334#endif
1335
1336// (ARM64EC defines _M_X64 and _M_AMD64 as well as _M_ARM64EC, so the x64 tests have to leave it out - it is NEON below)
1337#if ( defined(_x86_64) || defined( __x86_64__ ) || defined( _M_X64 ) || defined(__x86_64) || defined(_M_AMD64) || defined(__SSE2__) || defined(STBIR_SSE) || defined(STBIR_SSE2) ) && !defined(_M_ARM64EC)
1338 #ifndef STBIR_SSE2
1339 #define STBIR_SSE2
1340 #endif
1341 // AVX-512: the foundation plus the byte/word (BW) and 128/256-bit (VL) extensions. /arch:AVX512 on microsoft,
1342 // -mavx512f -mavx512bw -mavx512vl (or -march=x86-64-v4) on clang and gcc. Also implies AVX2
1343 #if ( defined(__AVX512F__) && defined(__AVX512BW__) && defined(__AVX512VL__) ) || defined(STBIR_AVX512)
1344 #ifdef STBIR_NO_AVX512
1345 #undef STBIR_AVX512
1346 #else
1347 #ifndef STBIR_AVX512
1348 #define STBIR_AVX512
1349 #endif
1350 #ifndef STBIR_AVX2
1351 #define STBIR_AVX2
1352 #endif
1353 #endif
1354 #endif
1355 #if defined(__AVX__) || defined(STBIR_AVX2)
1356 #ifndef STBIR_AVX
1357 #ifndef STBIR_NO_AVX
1358 #define STBIR_AVX
1359 #endif
1360 #endif
1361 #endif
1362 #if defined(__AVX2__) || defined(STBIR_AVX2)
1363 #ifndef STBIR_NO_AVX2
1364 #ifndef STBIR_AVX2
1365 #define STBIR_AVX2
1366 #endif
1367 #if defined( _MSC_VER ) && !defined(__clang__)
1368 #ifndef STBIR_FP16C // FP16C instructions are on all AVX2 cpus, so we can autoselect it here on microsoft - clang needs -mf16c
1369 #define STBIR_FP16C
1370 #endif
1371 #endif
1372 #endif
1373 #endif
1374 #ifdef __F16C__
1375 #ifndef STBIR_FP16C // turn on FP16C instructions if the define is set (for clang and gcc)
1376 #define STBIR_FP16C
1377 #endif
1378 #endif
1379#endif
1380
1381// 64-bit ARM (MSVC's ARM64EC included): the NEON code has a few AArch64 only forms
1382#if defined( _M_ARM64 ) || defined( _M_ARM64EC ) || defined( __aarch64__ ) || defined( __arm64__ )
1383#define STBIR__ARM64
1384#endif
1385
1386#if defined( STBIR__ARM64 ) || ((__ARM_NEON_FP & 4) != 0) || defined(__ARM_NEON__)
1387#ifndef STBIR_NEON
1388#define STBIR_NEON
1389#endif
1390#endif
1391
1392#if defined(_M_ARM) || defined(__arm__)
1393#ifdef STBIR_USE_FMA
1394#undef STBIR_USE_FMA // no FMA for 32-bit arm on MSVC
1395#endif
1396#endif
1397
1398#if defined(__wasm__) && defined(__wasm_simd128__)
1399#ifndef STBIR_WASM
1400#define STBIR_WASM
1401#endif
1402#endif
1403
1404// restrict pointers for the output pointers, other loop and unroll control
1405#if defined( _MSC_VER ) && !defined(__clang__)
1406 #define STBIR_STREAMOUT_PTR( star ) star __restrict
1407 #define STBIR_NO_UNROLL( ptr ) __assume(ptr) // this oddly keeps msvc from unrolling a loop
1408 #if _MSC_VER >= 1900
1409 #define STBIR_NO_UNROLL_LOOP_START __pragma(loop( no_vector ))
1410 #else
1411 #define STBIR_NO_UNROLL_LOOP_START
1412 #endif
1413#elif defined( __clang__ )
1414 #define STBIR_STREAMOUT_PTR( star ) star __restrict__
1415 #define STBIR_NO_UNROLL( ptr ) __asm__ (""::"r"(ptr))
1416 #if ( __clang_major__ >= 4 ) || ( ( __clang_major__ >= 3 ) && ( __clang_minor__ >= 5 ) )
1417 #define STBIR_NO_UNROLL_LOOP_START _Pragma("clang loop unroll(disable)") _Pragma("clang loop vectorize(disable)")
1418 #else
1419 #define STBIR_NO_UNROLL_LOOP_START
1420 #endif
1421#elif defined( __GNUC__ )
1422 #define STBIR_STREAMOUT_PTR( star ) star __restrict__
1423 #define STBIR_NO_UNROLL( ptr ) __asm__ (""::"r"(ptr))
1424 #if __GNUC__ >= 14
1425 #define STBIR_NO_UNROLL_LOOP_START _Pragma("GCC unroll 0") _Pragma("GCC novector")
1426 #else
1427 #define STBIR_NO_UNROLL_LOOP_START
1428 #endif
1429 #define STBIR_NO_UNROLL_LOOP_START_INF_FOR
1430#else
1431 #define STBIR_STREAMOUT_PTR( star ) star
1432 #define STBIR_NO_UNROLL( ptr )
1433 #define STBIR_NO_UNROLL_LOOP_START
1434#endif
1435
1436#ifndef STBIR_NO_UNROLL_LOOP_START_INF_FOR
1437#define STBIR_NO_UNROLL_LOOP_START_INF_FOR STBIR_NO_UNROLL_LOOP_START
1438#endif
1439
1440#ifdef STBIR_NO_SIMD // force simd off for whatever reason
1441
1442// force simd off overrides everything else, so clear it all
1443
1444#ifdef STBIR_SSE2
1445#undef STBIR_SSE2
1446#endif
1447
1448#ifdef STBIR_AVX
1449#undef STBIR_AVX
1450#endif
1451
1452#ifdef STBIR_NEON
1453#undef STBIR_NEON
1454#endif
1455
1456#ifdef STBIR_AVX2
1457#undef STBIR_AVX2
1458#endif
1459
1460#ifdef STBIR_FP16C
1461#undef STBIR_FP16C
1462#endif
1463
1464#ifdef STBIR_WASM
1465#undef STBIR_WASM
1466#endif
1467
1468#ifdef STBIR_SIMD
1469#undef STBIR_SIMD
1470#endif
1471
1472#else // STBIR_SIMD
1473
1474#ifdef STBIR_SSE2
1475 #include <emmintrin.h>
1476
1477 #define stbir__simdf __m128
1478 #define stbir__simdi __m128i
1479
1480 #define stbir_simdi_castf( reg ) _mm_castps_si128(reg)
1481 #define stbir_simdf_casti( reg ) _mm_castsi128_ps(reg)
1482
1483 #define stbir__simdf_load( reg, ptr ) (reg) = _mm_loadu_ps( (float const*)(ptr) )
1484 #define stbir__simdi_load( reg, ptr ) (reg) = _mm_loadu_si128 ( (stbir__simdi const*)(ptr) )
1485 #define stbir__simdf_load1( out, ptr ) (out) = _mm_load_ss( (float const*)(ptr) ) // top values can be random (not denormal or nan for perf)
1486 #define stbir__simdf_load1z( out, ptr ) (out) = _mm_load_ss( (float const*)(ptr) ) // top values must be zero
1487 #define stbir__simdf_frep4( fvar ) _mm_set_ps1( fvar )
1488 #define stbir__simdf_load1frep4( out, fvar ) (out) = _mm_set_ps1( fvar )
1489 #define stbir__simdf_load2( out, ptr ) (out) = _mm_castsi128_ps( _mm_loadl_epi64( (__m128i*)(ptr)) ) // top values can be random (not denormal or nan for perf)
1490 #define stbir__simdf_load2z( out, ptr ) (out) = _mm_castsi128_ps( _mm_loadl_epi64( (__m128i*)(ptr)) ) // top values must be zero
1491 #define stbir__simdf_load2hmerge( out, reg, ptr ) (out) = _mm_castpd_ps(_mm_loadh_pd( _mm_castps_pd(reg), (double*)(ptr) ))
1492 // pairwise add of two registers: out = ( a0+a1, a2+a3, b0+b1, b2+b3 ), and the halves version: ( a0+a2, a1+a3, b0+b2, b1+b3 )
1493 #define stbir__simdf_hadd2( out, a, b ) (out) = _mm_add_ps( _mm_shuffle_ps( a, b, (0<<0)+(2<<2)+(0<<4)+(2<<6) ), _mm_shuffle_ps( a, b, (1<<0)+(3<<2)+(1<<4)+(3<<6) ) )
1494 #define stbir__simdf_addhalves2( out, a, b ) (out) = _mm_add_ps( _mm_movelh_ps( a, b ), _mm_movehl_ps( b, a ) )
1495
1496 #define stbir__simdf_zeroP() _mm_setzero_ps()
1497 #define stbir__simdf_zero( reg ) (reg) = _mm_setzero_ps()
1498
1499 #define stbir__simdf_store( ptr, reg ) _mm_storeu_ps( (float*)(ptr), reg )
1500 #define stbir__simdf_store1( ptr, reg ) _mm_store_ss( (float*)(ptr), reg )
1501 #define stbir__simdf_store2( ptr, reg ) _mm_storel_epi64( (__m128i*)(ptr), _mm_castps_si128(reg) )
1502 #define stbir__simdf_store2h( ptr, reg ) _mm_storeh_pd( (double*)(ptr), _mm_castps_pd(reg) )
1503
1504 #define stbir__simdi_store( ptr, reg ) _mm_storeu_si128( (__m128i*)(ptr), reg )
1505 #define stbir__simdi_store2( ptr, reg ) _mm_storel_epi64( (__m128i*)(ptr), (reg) )
1506
1507 #define stbir__prefetch( ptr ) _mm_prefetch((char*)(ptr), _MM_HINT_T0 )
1508
1509 #define stbir__simdi_expand_u8_to_u32(out0,out1,out2,out3,ireg) \
1510 { \
1511 stbir__simdi zero = _mm_setzero_si128(); \
1512 out2 = _mm_unpacklo_epi8( ireg, zero ); \
1513 out3 = _mm_unpackhi_epi8( ireg, zero ); \
1514 out0 = _mm_unpacklo_epi16( out2, zero ); \
1515 out1 = _mm_unpackhi_epi16( out2, zero ); \
1516 out2 = _mm_unpacklo_epi16( out3, zero ); \
1517 out3 = _mm_unpackhi_epi16( out3, zero ); \
1518 }
1519
1520
1521 #define stbir__simdi_expand_u16_to_u32(out0,out1,ireg) \
1522 { \
1523 stbir__simdi zero = _mm_setzero_si128(); \
1524 out0 = _mm_unpacklo_epi16( ireg, zero ); \
1525 out1 = _mm_unpackhi_epi16( ireg, zero ); \
1526 }
1527
1528 #define stbir__simdf_convert_float_to_i32( i, f ) (i) = _mm_cvttps_epi32(f)
1529 #define stbir__simdf_convert_float_to_uint8( f ) ((unsigned char)_mm_cvtsi128_si32(_mm_cvttps_epi32(_mm_max_ps(_mm_min_ps(f,STBIR__CONSTF(STBIR_max_uint8_as_float)),_mm_setzero_ps()))))
1530 #define stbir__simdf_convert_float_to_short( f ) ((unsigned short)_mm_cvtsi128_si32(_mm_cvttps_epi32(_mm_max_ps(_mm_min_ps(f,STBIR__CONSTF(STBIR_max_uint16_as_float)),_mm_setzero_ps()))))
1531
1532 #define stbir__simdi_to_int( i ) _mm_cvtsi128_si32(i)
1533 #define stbir__simdi_convert_i32_to_float(out, ireg) (out) = _mm_cvtepi32_ps( ireg )
1534 #define stbir__simdf_add( out, reg0, reg1 ) (out) = _mm_add_ps( reg0, reg1 )
1535 #define stbir__simdf_mult( out, reg0, reg1 ) (out) = _mm_mul_ps( reg0, reg1 )
1536 #define stbir__simdf_mult_mem( out, reg, ptr ) (out) = _mm_mul_ps( reg, _mm_loadu_ps( (float const*)(ptr) ) )
1537 #define stbir__simdf_mult1_mem( out, reg, ptr ) (out) = _mm_mul_ss( reg, _mm_load_ss( (float const*)(ptr) ) )
1538 #define stbir__simdf_add_mem( out, reg, ptr ) (out) = _mm_add_ps( reg, _mm_loadu_ps( (float const*)(ptr) ) )
1539
1540 #ifdef STBIR_USE_FMA // not on by default to maintain bit identical simd to non-simd
1541 #include <immintrin.h>
1542 #define stbir__simdf_madd( out, add, mul1, mul2 ) (out) = _mm_fmadd_ps( mul1, mul2, add )
1543 #define stbir__simdf_madd_mem( out, add, mul, ptr ) (out) = _mm_fmadd_ps( mul, _mm_loadu_ps( (float const*)(ptr) ), add )
1544 #define stbir__simdf_madd1_mem( out, add, mul, ptr ) (out) = _mm_fmadd_ss( mul, _mm_load_ss( (float const*)(ptr) ), add )
1545 #else
1546 #define stbir__simdf_madd( out, add, mul1, mul2 ) (out) = _mm_add_ps( add, _mm_mul_ps( mul1, mul2 ) )
1547 #define stbir__simdf_madd_mem( out, add, mul, ptr ) (out) = _mm_add_ps( add, _mm_mul_ps( mul, _mm_loadu_ps( (float const*)(ptr) ) ) )
1548 #define stbir__simdf_madd1_mem( out, add, mul, ptr ) (out) = _mm_add_ss( add, _mm_mul_ss( mul, _mm_load_ss( (float const*)(ptr) ) ) )
1549 #endif
1550
1551 #define stbir__simdf_add1( out, reg0, reg1 ) (out) = _mm_add_ss( reg0, reg1 )
1552
1553 #define stbir__simdf_and( out, reg0, reg1 ) (out) = _mm_and_ps( reg0, reg1 )
1554
1555 #define stbir__simdf_min( out, reg0, reg1 ) (out) = _mm_min_ps( reg0, reg1 )
1556 #define stbir__simdf_max( out, reg0, reg1 ) (out) = _mm_max_ps( reg0, reg1 )
1557
1558 #define stbir__simdf_0123ABCDto3ABx( out, reg0, reg1 ) (out)=_mm_castsi128_ps( _mm_shuffle_epi32( _mm_castps_si128( _mm_shuffle_ps( reg1,reg0, (0<<0) + (1<<2) + (2<<4) + (3<<6) )), (3<<0) + (0<<2) + (1<<4) + (2<<6) ) )
1559 #define stbir__simdf_0123ABCDto23Ax( out, reg0, reg1 ) (out)=_mm_castsi128_ps( _mm_shuffle_epi32( _mm_castps_si128( _mm_shuffle_ps( reg1,reg0, (0<<0) + (1<<2) + (2<<4) + (3<<6) )), (2<<0) + (3<<2) + (0<<4) + (1<<6) ) )
1560
1561 static const stbir__simdf STBIR_zeroones = { 0.0f,1.0f,0.0f,1.0f };
1562 static const stbir__simdf STBIR_onezeros = { 1.0f,0.0f,1.0f,0.0f };
1563 #define stbir__simdf_aaa1( out, alp, ones ) (out)=_mm_castsi128_ps( _mm_shuffle_epi32( _mm_castps_si128( _mm_movehl_ps( ones, alp ) ), (1<<0) + (1<<2) + (1<<4) + (2<<6) ) )
1564 #define stbir__simdf_1aaa( out, alp, ones ) (out)=_mm_castsi128_ps( _mm_shuffle_epi32( _mm_castps_si128( _mm_movelh_ps( ones, alp ) ), (0<<0) + (2<<2) + (2<<4) + (2<<6) ) )
1565 #define stbir__simdf_a1a1( out, alp, ones) (out) = _mm_or_ps( _mm_castsi128_ps( _mm_srli_epi64( _mm_castps_si128(alp), 32 ) ), STBIR_zeroones )
1566 #define stbir__simdf_1a1a( out, alp, ones) (out) = _mm_or_ps( _mm_castsi128_ps( _mm_slli_epi64( _mm_castps_si128(alp), 32 ) ), STBIR_onezeros )
1567
1568 #define stbir__simdf_swiz( reg, one, two, three, four ) _mm_castsi128_ps( _mm_shuffle_epi32( _mm_castps_si128( reg ), (one<<0) + (two<<2) + (three<<4) + (four<<6) ) )
1569
1570 #define stbir__simdi_and( out, reg0, reg1 ) (out) = _mm_and_si128( reg0, reg1 )
1571 #define stbir__simdi_or( out, reg0, reg1 ) (out) = _mm_or_si128( reg0, reg1 )
1572 #define stbir__simdi_16madd( out, reg0, reg1 ) (out) = _mm_madd_epi16( reg0, reg1 )
1573
1574 #define stbir__simdf_pack_to_8bytes(out,aa,bb) \
1575 { \
1576 stbir__simdf af,bf; \
1577 stbir__simdi a,b; \
1578 af = _mm_min_ps( aa, STBIR_max_uint8_as_float ); \
1579 bf = _mm_min_ps( bb, STBIR_max_uint8_as_float ); \
1580 af = _mm_max_ps( af, _mm_setzero_ps() ); \
1581 bf = _mm_max_ps( bf, _mm_setzero_ps() ); \
1582 a = _mm_cvttps_epi32( af ); \
1583 b = _mm_cvttps_epi32( bf ); \
1584 a = _mm_packs_epi32( a, b ); \
1585 out = _mm_packus_epi16( a, a ); \
1586 }
1587
1588 #define stbir__simdf_load4_transposed( o0, o1, o2, o3, ptr ) \
1589 stbir__simdf_load( o0, (ptr) ); \
1590 stbir__simdf_load( o1, (ptr)+4 ); \
1591 stbir__simdf_load( o2, (ptr)+8 ); \
1592 stbir__simdf_load( o3, (ptr)+12 ); \
1593 { \
1594 __m128 tmp0, tmp1, tmp2, tmp3; \
1595 tmp0 = _mm_unpacklo_ps(o0, o1); \
1596 tmp2 = _mm_unpacklo_ps(o2, o3); \
1597 tmp1 = _mm_unpackhi_ps(o0, o1); \
1598 tmp3 = _mm_unpackhi_ps(o2, o3); \
1599 o0 = _mm_movelh_ps(tmp0, tmp2); \
1600 o1 = _mm_movehl_ps(tmp2, tmp0); \
1601 o2 = _mm_movelh_ps(tmp1, tmp3); \
1602 o3 = _mm_movehl_ps(tmp3, tmp1); \
1603 }
1604
1605 #define stbir__interleave_pack_and_store_16_u8( ptr, r0, r1, r2, r3 ) \
1606 r0 = _mm_packs_epi32( r0, r1 ); \
1607 r2 = _mm_packs_epi32( r2, r3 ); \
1608 r1 = _mm_unpacklo_epi16( r0, r2 ); \
1609 r3 = _mm_unpackhi_epi16( r0, r2 ); \
1610 r0 = _mm_unpacklo_epi16( r1, r3 ); \
1611 r2 = _mm_unpackhi_epi16( r1, r3 ); \
1612 r0 = _mm_packus_epi16( r0, r2 ); \
1613 stbir__simdi_store( ptr, r0 ); \
1614
1615 #define stbir__simdi_32shr( out, reg, imm ) out = _mm_srli_epi32( reg, imm )
1616
1617 #if defined(_MSC_VER) && !defined(__clang__)
1618 // msvc inits with 8 bytes
1619 #define STBIR__CONST_32_TO_8( v ) (char)(unsigned char)((v)&255),(char)(unsigned char)(((v)>>8)&255),(char)(unsigned char)(((v)>>16)&255),(char)(unsigned char)(((v)>>24)&255)
1620 #define STBIR__CONST_4_32i( v ) STBIR__CONST_32_TO_8( v ), STBIR__CONST_32_TO_8( v ), STBIR__CONST_32_TO_8( v ), STBIR__CONST_32_TO_8( v )
1621 #define STBIR__CONST_4d_32i( v0, v1, v2, v3 ) STBIR__CONST_32_TO_8( v0 ), STBIR__CONST_32_TO_8( v1 ), STBIR__CONST_32_TO_8( v2 ), STBIR__CONST_32_TO_8( v3 )
1622 #else
1623 // everything else inits with long long's
1624 #define STBIR__CONST_4_32i( v ) (long long)((((stbir_uint64)(stbir_uint32)(v))<<32)|((stbir_uint64)(stbir_uint32)(v))),(long long)((((stbir_uint64)(stbir_uint32)(v))<<32)|((stbir_uint64)(stbir_uint32)(v)))
1625 #define STBIR__CONST_4d_32i( v0, v1, v2, v3 ) (long long)((((stbir_uint64)(stbir_uint32)(v1))<<32)|((stbir_uint64)(stbir_uint32)(v0))),(long long)((((stbir_uint64)(stbir_uint32)(v3))<<32)|((stbir_uint64)(stbir_uint32)(v2)))
1626 #endif
1627
1628 #define STBIR__SIMDF_CONST(var, x) stbir__simdf var = { x, x, x, x }
1629 #define STBIR__SIMDI_CONST(var, x) stbir__simdi var = { STBIR__CONST_4_32i(x) }
1630 #define STBIR__SIMDI_CONST4(var, a, b, c, d) stbir__simdi var = { STBIR__CONST_4d_32i(a, b, c, d) }
1631 #define STBIR__CONSTF(var) (var)
1632 #define STBIR__CONSTI(var) (var)
1633
1634 // an arbitrary byte permute (pshufb) needs SSSE3
1635 #if defined(STBIR_AVX) || defined(__SSSE3__) || defined(__SSE4_1__)
1636 #include <tmmintrin.h>
1637 #define STBIR_SIMD_BYTE_SHUFFLE
1638 #define stbir__simdi_shuffle_bytes( out, reg, table ) (out) = _mm_shuffle_epi8( (reg), (table) )
1639 #endif
1640
1641 #if defined(STBIR_AVX) || defined(__SSE4_1__)
1642 #include <smmintrin.h>
1643 #define stbir__simdf_pack_to_8words(out,reg0,reg1) out = _mm_packus_epi32(_mm_cvttps_epi32(_mm_max_ps(_mm_min_ps(reg0,STBIR__CONSTF(STBIR_max_uint16_as_float)),_mm_setzero_ps())), _mm_cvttps_epi32(_mm_max_ps(_mm_min_ps(reg1,STBIR__CONSTF(STBIR_max_uint16_as_float)),_mm_setzero_ps())))
1644 #else
1645 static STBIR__SIMDI_CONST(stbir__s32_32768, 32768);
1646 static STBIR__SIMDI_CONST(stbir__s16_32768, ((32768<<16)|32768));
1647
1648 #define stbir__simdf_pack_to_8words(out,reg0,reg1) \
1649 { \
1650 stbir__simdi tmp0,tmp1; \
1651 tmp0 = _mm_cvttps_epi32(_mm_max_ps(_mm_min_ps(reg0,STBIR__CONSTF(STBIR_max_uint16_as_float)),_mm_setzero_ps())); \
1652 tmp1 = _mm_cvttps_epi32(_mm_max_ps(_mm_min_ps(reg1,STBIR__CONSTF(STBIR_max_uint16_as_float)),_mm_setzero_ps())); \
1653 tmp0 = _mm_sub_epi32( tmp0, stbir__s32_32768 ); \
1654 tmp1 = _mm_sub_epi32( tmp1, stbir__s32_32768 ); \
1655 out = _mm_packs_epi32( tmp0, tmp1 ); \
1656 out = _mm_sub_epi16( out, stbir__s16_32768 ); \
1657 }
1658
1659 #endif
1660
1661 #define STBIR_SIMD
1662
1663 // if we detect AVX, set the simd8 defines
1664 #ifdef STBIR_AVX
1665 #include <immintrin.h>
1666 #define STBIR_SIMD8
1667 #define stbir__simdf8 __m256
1668 #define stbir__simdi8 __m256i
1669 #define stbir__simdf8_load( out, ptr ) (out) = _mm256_loadu_ps( (float const *)(ptr) )
1670 #define stbir__simdi8_load( out, ptr ) (out) = _mm256_loadu_si256( (__m256i const *)(ptr) )
1671 #define stbir__simdf8_mult( out, a, b ) (out) = _mm256_mul_ps( (a), (b) )
1672 #define stbir__simdf8_store( ptr, out ) _mm256_storeu_ps( (float*)(ptr), out )
1673 #define stbir__simdi8_store( ptr, reg ) _mm256_storeu_si256( (__m256i*)(ptr), reg )
1674 #define stbir__simdf8_frep8( fval ) _mm256_set1_ps( fval )
1675
1676 #define stbir__simdf8_min( out, reg0, reg1 ) (out) = _mm256_min_ps( reg0, reg1 )
1677 #define stbir__simdf8_max( out, reg0, reg1 ) (out) = _mm256_max_ps( reg0, reg1 )
1678
1679 #define stbir__simdf8_add4halves( out, bot4, top8 ) (out) = _mm_add_ps( bot4, _mm256_extractf128_ps( top8, 1 ) )
1680 #define stbir__simdf8_mult_mem( out, reg, ptr ) (out) = _mm256_mul_ps( reg, _mm256_loadu_ps( (float const*)(ptr) ) )
1681 #define stbir__simdf8_add_mem( out, reg, ptr ) (out) = _mm256_add_ps( reg, _mm256_loadu_ps( (float const*)(ptr) ) )
1682 #define stbir__simdf8_add( out, a, b ) (out) = _mm256_add_ps( a, b )
1683 #define stbir__simdf8_load1b( out, ptr ) (out) = _mm256_broadcast_ss( ptr )
1684 #define stbir__simdf_load1rep4( out, ptr ) (out) = _mm_broadcast_ss( ptr ) // avx load instruction
1685
1686 #define stbir__simdi8_convert_i32_to_float(out, ireg) (out) = _mm256_cvtepi32_ps( ireg )
1687 #define stbir__simdf8_convert_float_to_i32( i, f ) (i) = _mm256_cvttps_epi32(f)
1688
1689 #define stbir__simdf8_bot4s( out, a, b ) (out) = _mm256_permute2f128_ps(a,b, (0<<0)+(2<<4) )
1690 #define stbir__simdf8_top4s( out, a, b ) (out) = _mm256_permute2f128_ps(a,b, (1<<0)+(3<<4) )
1691
1692 #define stbir__simdf8_gettop4( reg ) _mm256_extractf128_ps(reg,1)
1693
1694 #ifdef STBIR_AVX2
1695
1696 // vpmovzxbd takes eight bytes to eight dwords in one shuffle, so sixteen bytes are three (the high eight moved down first)
1697 #define stbir__simdi8_expand_u8_to_u32(out0,out1,ireg) \
1698 { \
1699 out0 = _mm256_cvtepu8_epi32( ireg ); \
1700 out1 = _mm256_cvtepu8_epi32( _mm_unpackhi_epi64( ireg, ireg ) ); \
1701 }
1702
1703 #define stbir__simdf8_pack_to_16bytes(out,aa,bb) \
1704 { \
1705 stbir__simdi8 t; \
1706 stbir__simdf8 af,bf; \
1707 stbir__simdi8 a,b; \
1708 af = _mm256_min_ps( aa, STBIR_max_uint8_as_float8 ); \
1709 bf = _mm256_min_ps( bb, STBIR_max_uint8_as_float8 ); \
1710 af = _mm256_max_ps( af, _mm256_setzero_ps() ); \
1711 bf = _mm256_max_ps( bf, _mm256_setzero_ps() ); \
1712 a = _mm256_cvttps_epi32( af ); \
1713 b = _mm256_cvttps_epi32( bf ); \
1714 t = _mm256_permute4x64_epi64( _mm256_packs_epi32( a, b ), (0<<0)+(2<<2)+(1<<4)+(3<<6) ); \
1715 out = _mm256_castsi256_si128( _mm256_permute4x64_epi64( _mm256_packus_epi16( t, t ), (0<<0)+(2<<2)+(1<<4)+(3<<6) ) ); \
1716 }
1717
1718 #define stbir__simdi8_expand_u16_to_u32(out,ireg) out = _mm256_cvtepu16_epi32( ireg );
1719
1720 // a byte permute of 32 bytes by one 16 byte table, per 128-bit lane (vpshufb ymm)
1721 #define stbir__simdi8_shuffle_bytes( out, reg, table ) (out) = _mm256_shuffle_epi8( (reg), _mm256_broadcastsi128_si256( table ) )
1722
1723 #define stbir__simdf8_pack_to_16words(out,aa,bb) \
1724 { \
1725 stbir__simdf8 af,bf; \
1726 stbir__simdi8 a,b; \
1727 af = _mm256_min_ps( aa, STBIR_max_uint16_as_float8 ); \
1728 bf = _mm256_min_ps( bb, STBIR_max_uint16_as_float8 ); \
1729 af = _mm256_max_ps( af, _mm256_setzero_ps() ); \
1730 bf = _mm256_max_ps( bf, _mm256_setzero_ps() ); \
1731 a = _mm256_cvttps_epi32( af ); \
1732 b = _mm256_cvttps_epi32( bf ); \
1733 (out) = _mm256_permute4x64_epi64( _mm256_packus_epi32(a, b), (0<<0)+(2<<2)+(1<<4)+(3<<6) ); \
1734 }
1735
1736 #else
1737
1738 #define stbir__simdi8_expand_u8_to_u32(out0,out1,ireg) \
1739 { \
1740 stbir__simdi a,zero = _mm_setzero_si128(); \
1741 a = _mm_unpacklo_epi8( ireg, zero ); \
1742 out0 = _mm256_setr_m128i( _mm_unpacklo_epi16( a, zero ), _mm_unpackhi_epi16( a, zero ) ); \
1743 a = _mm_unpackhi_epi8( ireg, zero ); \
1744 out1 = _mm256_setr_m128i( _mm_unpacklo_epi16( a, zero ), _mm_unpackhi_epi16( a, zero ) ); \
1745 }
1746
1747 #define stbir__simdf8_pack_to_16bytes(out,aa,bb) \
1748 { \
1749 stbir__simdi t; \
1750 stbir__simdf8 af,bf; \
1751 stbir__simdi8 a,b; \
1752 af = _mm256_min_ps( aa, STBIR_max_uint8_as_float8 ); \
1753 bf = _mm256_min_ps( bb, STBIR_max_uint8_as_float8 ); \
1754 af = _mm256_max_ps( af, _mm256_setzero_ps() ); \
1755 bf = _mm256_max_ps( bf, _mm256_setzero_ps() ); \
1756 a = _mm256_cvttps_epi32( af ); \
1757 b = _mm256_cvttps_epi32( bf ); \
1758 out = _mm_packs_epi32( _mm256_castsi256_si128(a), _mm256_extractf128_si256( a, 1 ) ); \
1759 out = _mm_packus_epi16( out, out ); \
1760 t = _mm_packs_epi32( _mm256_castsi256_si128(b), _mm256_extractf128_si256( b, 1 ) ); \
1761 t = _mm_packus_epi16( t, t ); \
1762 out = _mm_castps_si128( _mm_shuffle_ps( _mm_castsi128_ps(out), _mm_castsi128_ps(t), (0<<0)+(1<<2)+(0<<4)+(1<<6) ) ); \
1763 }
1764
1765 #define stbir__simdi8_expand_u16_to_u32(out,ireg) \
1766 { \
1767 stbir__simdi a,b,zero = _mm_setzero_si128(); \
1768 a = _mm_unpacklo_epi16( ireg, zero ); \
1769 b = _mm_unpackhi_epi16( ireg, zero ); \
1770 out = _mm256_insertf128_si256( _mm256_castsi128_si256( a ), b, 1 ); \
1771 }
1772
1773 #define stbir__simdf8_pack_to_16words(out,aa,bb) \
1774 { \
1775 stbir__simdi t0,t1; \
1776 stbir__simdf8 af,bf; \
1777 stbir__simdi8 a,b; \
1778 af = _mm256_min_ps( aa, STBIR_max_uint16_as_float8 ); \
1779 bf = _mm256_min_ps( bb, STBIR_max_uint16_as_float8 ); \
1780 af = _mm256_max_ps( af, _mm256_setzero_ps() ); \
1781 bf = _mm256_max_ps( bf, _mm256_setzero_ps() ); \
1782 a = _mm256_cvttps_epi32( af ); \
1783 b = _mm256_cvttps_epi32( bf ); \
1784 t0 = _mm_packus_epi32( _mm256_castsi256_si128(a), _mm256_extractf128_si256( a, 1 ) ); \
1785 t1 = _mm_packus_epi32( _mm256_castsi256_si128(b), _mm256_extractf128_si256( b, 1 ) ); \
1786 out = _mm256_setr_m128i( t0, t1 ); \
1787 }
1788
1789 #endif
1790
1791 static __m256i stbir_00001111 = { STBIR__CONST_4d_32i( 0, 0, 0, 0 ), STBIR__CONST_4d_32i( 1, 1, 1, 1 ) };
1792 #define stbir__simdf8_0123to00001111( out, in ) (out) = _mm256_permutevar_ps ( in, stbir_00001111 )
1793
1794 static __m256i stbir_22223333 = { STBIR__CONST_4d_32i( 2, 2, 2, 2 ), STBIR__CONST_4d_32i( 3, 3, 3, 3 ) };
1795 #define stbir__simdf8_0123to22223333( out, in ) (out) = _mm256_permutevar_ps ( in, stbir_22223333 )
1796
1797 #define stbir__simdf8_0123to2222( out, in ) (out) = stbir__simdf_swiz(_mm256_castps256_ps128(in), 2,2,2,2 )
1798
1799 #define stbir__simdf8_load4b( out, ptr ) (out) = _mm256_broadcast_ps( (__m128 const *)(ptr) )
1800
1801 static __m256i stbir_00112233 = { STBIR__CONST_4d_32i( 0, 0, 1, 1 ), STBIR__CONST_4d_32i( 2, 2, 3, 3 ) };
1802 #define stbir__simdf8_0123to00112233( out, in ) (out) = _mm256_permutevar_ps ( in, stbir_00112233 )
1803 #define stbir__simdf8_add4( out, a8, b ) (out) = _mm256_add_ps( a8, _mm256_insertf128_ps( _mm256_setzero_ps(), b, 0 ) )
1804
1805 static __m256i stbir_load6 = { STBIR__CONST_4_32i( 0x80000000 ), STBIR__CONST_4d_32i( 0x80000000, 0x80000000, 0, 0 ) };
1806 #define stbir__simdf8_load6z( out, ptr ) (out) = _mm256_maskload_ps( ptr, stbir_load6 )
1807
1808 #define stbir__simdf8_0123to00000000( out, in ) (out) = _mm256_shuffle_ps ( in, in, (0<<0)+(0<<2)+(0<<4)+(0<<6) )
1809 #define stbir__simdf8_0123to11111111( out, in ) (out) = _mm256_shuffle_ps ( in, in, (1<<0)+(1<<2)+(1<<4)+(1<<6) )
1810 #define stbir__simdf8_0123to22222222( out, in ) (out) = _mm256_shuffle_ps ( in, in, (2<<0)+(2<<2)+(2<<4)+(2<<6) )
1811 #define stbir__simdf8_0123to33333333( out, in ) (out) = _mm256_shuffle_ps ( in, in, (3<<0)+(3<<2)+(3<<4)+(3<<6) )
1812 #define stbir__simdf8_0123to21032103( out, in ) (out) = _mm256_shuffle_ps ( in, in, (2<<0)+(1<<2)+(0<<4)+(3<<6) )
1813 #define stbir__simdf8_0123to32103210( out, in ) (out) = _mm256_shuffle_ps ( in, in, (3<<0)+(2<<2)+(1<<4)+(0<<6) )
1814 #define stbir__simdf8_0123to12301230( out, in ) (out) = _mm256_shuffle_ps ( in, in, (1<<0)+(2<<2)+(3<<4)+(0<<6) )
1815 #define stbir__simdf8_0123to10321032( out, in ) (out) = _mm256_shuffle_ps ( in, in, (1<<0)+(0<<2)+(3<<4)+(2<<6) )
1816 #define stbir__simdf8_0123to30123012( out, in ) (out) = _mm256_shuffle_ps ( in, in, (3<<0)+(0<<2)+(1<<4)+(2<<6) )
1817
1818 #define stbir__simdf8_0123to11331133( out, in ) (out) = _mm256_shuffle_ps ( in, in, (1<<0)+(1<<2)+(3<<4)+(3<<6) )
1819 #define stbir__simdf8_0123to00220022( out, in ) (out) = _mm256_shuffle_ps ( in, in, (0<<0)+(0<<2)+(2<<4)+(2<<6) )
1820
1821 #define stbir__simdf8_aaa1( out, alp, ones ) do { (out) = _mm256_blend_ps( alp, ones, (1<<0)+(1<<1)+(1<<2)+(0<<3)+(1<<4)+(1<<5)+(1<<6)+(0<<7)); (out)=_mm256_shuffle_ps( out,out, (3<<0) + (3<<2) + (3<<4) + (0<<6) ); } while(0)
1822 #define stbir__simdf8_1aaa( out, alp, ones ) do { (out) = _mm256_blend_ps( alp, ones, (0<<0)+(1<<1)+(1<<2)+(1<<3)+(0<<4)+(1<<5)+(1<<6)+(1<<7)); (out)=_mm256_shuffle_ps( out,out, (1<<0) + (0<<2) + (0<<4) + (0<<6) ); } while(0)
1823 #define stbir__simdf8_a1a1( out, alp, ones) do { (out) = _mm256_blend_ps( alp, ones, (1<<0)+(0<<1)+(1<<2)+(0<<3)+(1<<4)+(0<<5)+(1<<6)+(0<<7)); (out)=_mm256_shuffle_ps( out,out, (1<<0) + (0<<2) + (3<<4) + (2<<6) ); } while(0)
1824 #define stbir__simdf8_1a1a( out, alp, ones) do { (out) = _mm256_blend_ps( alp, ones, (0<<0)+(1<<1)+(0<<2)+(1<<3)+(0<<4)+(1<<5)+(0<<6)+(1<<7)); (out)=_mm256_shuffle_ps( out,out, (1<<0) + (0<<2) + (3<<4) + (2<<6) ); } while(0)
1825
1826 #define stbir__simdf8_zero( reg ) (reg) = _mm256_setzero_ps()
1827
1828 #ifdef STBIR_USE_FMA // not on by default to maintain bit identical simd to non-simd
1829 #define stbir__simdf8_madd( out, add, mul1, mul2 ) (out) = _mm256_fmadd_ps( mul1, mul2, add )
1830 #define stbir__simdf8_madd_mem( out, add, mul, ptr ) (out) = _mm256_fmadd_ps( mul, _mm256_loadu_ps( (float const*)(ptr) ), add )
1831 #define stbir__simdf8_madd_mem4( out, add, mul, ptr )(out) = _mm256_fmadd_ps( _mm256_setr_m128( mul, _mm_setzero_ps() ), _mm256_setr_m128( _mm_loadu_ps( (float const*)(ptr) ), _mm_setzero_ps() ), add )
1832 #else
1833 #define stbir__simdf8_madd( out, add, mul1, mul2 ) (out) = _mm256_add_ps( add, _mm256_mul_ps( mul1, mul2 ) )
1834 #define stbir__simdf8_madd_mem( out, add, mul, ptr ) (out) = _mm256_add_ps( add, _mm256_mul_ps( mul, _mm256_loadu_ps( (float const*)(ptr) ) ) )
1835 #define stbir__simdf8_madd_mem4( out, add, mul, ptr ) (out) = _mm256_add_ps( add, _mm256_setr_m128( _mm_mul_ps( mul, _mm_loadu_ps( (float const*)(ptr) ) ), _mm_setzero_ps() ) )
1836 #endif
1837 #define stbir__if_simdf8_cast_to_simdf4( val ) _mm256_castps256_ps128( val )
1838
1839 static const stbir__simdf8 STBIR_max_uint16_as_float_inverted8 = { stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted };
1840 static const stbir__simdf8 STBIR_max_uint8_as_float_inverted8 = { stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted };
1841 static const stbir__simdf8 STBIR_ones8 = { 1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0 };
1842 static const stbir__simdf8 STBIR_simd_point58 = { 0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5 };
1843 static const stbir__simdf8 STBIR_max_uint8_as_float8 = { stbir__max_uint8_as_float,stbir__max_uint8_as_float,stbir__max_uint8_as_float,stbir__max_uint8_as_float, stbir__max_uint8_as_float,stbir__max_uint8_as_float,stbir__max_uint8_as_float,stbir__max_uint8_as_float };
1844 static const stbir__simdf8 STBIR_max_uint16_as_float8 = { stbir__max_uint16_as_float,stbir__max_uint16_as_float,stbir__max_uint16_as_float,stbir__max_uint16_as_float, stbir__max_uint16_as_float,stbir__max_uint16_as_float,stbir__max_uint16_as_float,stbir__max_uint16_as_float };
1845
1846 // the 16-wide set (AVX-512) for the X-wide code.
1847 #ifdef STBIR_AVX512
1848 #define STBIR_SIMD16
1849 #define stbir__simdf16 __m512
1850 #define stbir__simdi16 __m512i
1851 #define stbir__simdf16_load( out, ptr ) (out) = _mm512_loadu_ps( (float const *)(ptr) )
1852 #define stbir__simdi16_load( out, ptr ) (out) = _mm512_loadu_si512( (void const *)(ptr) )
1853 #define stbir__simdf16_store( ptr, reg ) _mm512_storeu_ps( (float*)(ptr), reg )
1854 #define stbir__simdi16_store( ptr, reg ) _mm512_storeu_si512( (void*)(ptr), reg )
1855 #define stbir__simdf16_mult( out, a, b ) (out) = _mm512_mul_ps( (a), (b) )
1856 #define stbir__simdf16_add_mem( out, reg, ptr ) (out) = _mm512_add_ps( reg, _mm512_loadu_ps( (float const*)(ptr) ) )
1857 #define stbir__simdf16_min( out, a, b ) (out) = _mm512_min_ps( a, b )
1858 #define stbir__simdf16_max( out, a, b ) (out) = _mm512_max_ps( a, b )
1859 #define stbir__simdf16_zero( reg ) (reg) = _mm512_setzero_ps()
1860 #define stbir__simdf16_frep16( fval ) _mm512_set1_ps( fval )
1861 #define stbir__simdi16_convert_i32_to_float( out, ireg ) (out) = _mm512_cvtepi32_ps( ireg )
1862 #define stbir__simdf16_convert_float_to_i32( i, f ) (i) = _mm512_cvttps_epi32( f )
1863 #define stbir__simdf16_cast_to_simdf4( val ) _mm512_castps512_ps128( val )
1864
1865 #ifdef STBIR_USE_FMA // not on by default to maintain bit identical simd to non-simd
1866 #define stbir__simdf16_madd( out, add, mul1, mul2 ) (out) = _mm512_fmadd_ps( mul1, mul2, add )
1867 #define stbir__simdf16_madd_mem( out, add, mul, ptr ) (out) = _mm512_fmadd_ps( mul, _mm512_loadu_ps( (float const*)(ptr) ), add )
1868 #else
1869 #define stbir__simdf16_madd( out, add, mul1, mul2 ) (out) = _mm512_add_ps( add, _mm512_mul_ps( mul1, mul2 ) )
1870 #define stbir__simdf16_madd_mem( out, add, mul, ptr ) (out) = _mm512_add_ps( add, _mm512_mul_ps( mul, _mm512_loadu_ps( (float const*)(ptr) ) ) )
1871 #endif
1872
1873 // the per lane shuffles: one 4-wide order, applied in every lane
1874 #define stbir__simdf16_0123to0123( out, in ) (out) = (in)
1875 #define stbir__simdf16_0123to1230( out, in ) (out) = _mm512_shuffle_ps( in, in, (1<<0)+(2<<2)+(3<<4)+(0<<6) )
1876 #define stbir__simdf16_0123to2103( out, in ) (out) = _mm512_shuffle_ps( in, in, (2<<0)+(1<<2)+(0<<4)+(3<<6) )
1877 #define stbir__simdf16_0123to3210( out, in ) (out) = _mm512_shuffle_ps( in, in, (3<<0)+(2<<2)+(1<<4)+(0<<6) )
1878 #define stbir__simdf16_0123to1032( out, in ) (out) = _mm512_shuffle_ps( in, in, (1<<0)+(0<<2)+(3<<4)+(2<<6) )
1879 #define stbir__simdf16_0123to3012( out, in ) (out) = _mm512_shuffle_ps( in, in, (3<<0)+(0<<2)+(1<<4)+(2<<6) )
1880
1881 // the alpha patterns: blend the ones in (the mask repeats per lane), then shuffle each lane
1882 #define stbir__simdf16_aaa1( out, alp, ones ) do { (out) = _mm512_mask_blend_ps( (__mmask16)0x7777, alp, ones ); (out) = _mm512_shuffle_ps( out, out, (3<<0)+(3<<2)+(3<<4)+(0<<6) ); } while(0)
1883 #define stbir__simdf16_1aaa( out, alp, ones ) do { (out) = _mm512_mask_blend_ps( (__mmask16)0xeeee, alp, ones ); (out) = _mm512_shuffle_ps( out, out, (1<<0)+(0<<2)+(0<<4)+(0<<6) ); } while(0)
1884 #define stbir__simdf16_a1a1( out, alp, ones ) do { (out) = _mm512_mask_blend_ps( (__mmask16)0x5555, alp, ones ); (out) = _mm512_shuffle_ps( out, out, (1<<0)+(0<<2)+(3<<4)+(2<<6) ); } while(0)
1885 #define stbir__simdf16_1a1a( out, alp, ones ) do { (out) = _mm512_mask_blend_ps( (__mmask16)0xaaaa, alp, ones ); (out) = _mm512_shuffle_ps( out, out, (1<<0)+(0<<2)+(3<<4)+(2<<6) ); } while(0)
1886
1887 // integer conversions: sixteen bytes or words to sixteen dwords in one op, and the saturating narrows back
1888 #define stbir__simdi16_expand_u8_to_u32( out, ireg ) (out) = _mm512_cvtepu8_epi32( ireg )
1889 #define stbir__simdi16_expand_u16_to_u32( out, ireg ) (out) = _mm512_cvtepu16_epi32( ireg )
1890
1891 #define stbir__simdf16_pack_to_32bytes( out, aa, bb ) \
1892 { \
1893 stbir__simdf16 af, bf; \
1894 af = _mm512_min_ps( aa, STBIR_max_uint8_as_float16 ); \
1895 bf = _mm512_min_ps( bb, STBIR_max_uint8_as_float16 ); \
1896 af = _mm512_max_ps( af, _mm512_setzero_ps() ); \
1897 bf = _mm512_max_ps( bf, _mm512_setzero_ps() ); \
1898 (out) = _mm256_inserti128_si256( _mm256_castsi128_si256( _mm512_cvtusepi32_epi8( _mm512_cvttps_epi32( af ) ) ), _mm512_cvtusepi32_epi8( _mm512_cvttps_epi32( bf ) ), 1 ); \
1899 }
1900
1901 // the pack interleaves the two inputs per 128-bit lane, the 64-bit permute puts them back in order
1902 static const stbir__simdi16 stbir__pack_lanes_in_order16 = { STBIR__CONST_4d_32i( 0,0, 2,0 ), STBIR__CONST_4d_32i( 4,0, 6,0 ), STBIR__CONST_4d_32i( 1,0, 3,0 ), STBIR__CONST_4d_32i( 5,0, 7,0 ) }; // 64-bit indexes 0 2 4 6 1 3 5 7
1903 #define stbir__simdf16_pack_to_32words( out, aa, bb ) \
1904 { \
1905 stbir__simdf16 af, bf; \
1906 af = _mm512_min_ps( aa, STBIR_max_uint16_as_float16 ); \
1907 bf = _mm512_min_ps( bb, STBIR_max_uint16_as_float16 ); \
1908 af = _mm512_max_ps( af, _mm512_setzero_ps() ); \
1909 bf = _mm512_max_ps( bf, _mm512_setzero_ps() ); \
1910 (out) = _mm512_permutexvar_epi64( stbir__pack_lanes_in_order16, _mm512_packus_epi32( _mm512_cvttps_epi32( af ), _mm512_cvttps_epi32( bf ) ) ); \
1911 }
1912
1913 static const stbir__simdf16 STBIR_ones16 = { 1.0f,1.0f,1.0f,1.0f, 1.0f,1.0f,1.0f,1.0f, 1.0f,1.0f,1.0f,1.0f, 1.0f,1.0f,1.0f,1.0f };
1914 static const stbir__simdf16 STBIR_simd_point516 = { 0.5f,0.5f,0.5f,0.5f, 0.5f,0.5f,0.5f,0.5f, 0.5f,0.5f,0.5f,0.5f, 0.5f,0.5f,0.5f,0.5f };
1915 static const stbir__simdf16 STBIR_max_uint8_as_float16 = { 255.0f,255.0f,255.0f,255.0f, 255.0f,255.0f,255.0f,255.0f, 255.0f,255.0f,255.0f,255.0f, 255.0f,255.0f,255.0f,255.0f };
1916 static const stbir__simdf16 STBIR_max_uint16_as_float16 = { 65535.0f,65535.0f,65535.0f,65535.0f, 65535.0f,65535.0f,65535.0f,65535.0f, 65535.0f,65535.0f,65535.0f,65535.0f, 65535.0f,65535.0f,65535.0f,65535.0f };
1917 static const stbir__simdf16 STBIR_max_uint8_as_float_inverted16 = { stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted, stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted, stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted, stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted };
1918 static const stbir__simdf16 STBIR_max_uint16_as_float_inverted16 = { stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted, stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted, stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted, stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted };
1919 #endif // STBIR_AVX512
1920
1921 #endif
1922
1923#elif defined(STBIR_NEON)
1924
1925 #include <arm_neon.h>
1926
1927 #define stbir__simdf float32x4_t
1928 #define stbir__simdi uint32x4_t
1929
1930 #define stbir_simdi_castf( reg ) vreinterpretq_u32_f32(reg)
1931 #define stbir_simdf_casti( reg ) vreinterpretq_f32_u32(reg)
1932
1933 #define stbir__simdf_load( reg, ptr ) (reg) = vld1q_f32( (float const*)(ptr) )
1934 #define stbir__simdi_load( reg, ptr ) (reg) = vld1q_u32( (uint32_t const*)(ptr) )
1935 #define stbir__simdf_load1( out, ptr ) (out) = vld1q_lane_f32( (float const*)(ptr), vdupq_n_f32(0), 0 ) // top values zero, like SSE (a replicated inf would turn into nan against the zero coefficient lanes)
1936 #define stbir__simdf_load1z( out, ptr ) (out) = vld1q_lane_f32( (float const*)(ptr), vdupq_n_f32(0), 0 ) // top values must be zero
1937 #define stbir__simdf_frep4( fvar ) vdupq_n_f32( fvar )
1938 #define stbir__simdf_load1frep4( out, fvar ) (out) = vdupq_n_f32( fvar )
1939 #define stbir__simdf_load2( out, ptr ) (out) = vcombine_f32( vld1_f32( (float const*)(ptr) ), vcreate_f32(0) ) // top values can be random (not denormal or nan for perf)
1940 #define stbir__simdf_load2z( out, ptr ) (out) = vcombine_f32( vld1_f32( (float const*)(ptr) ), vcreate_f32(0) ) // top values must be zero
1941 #define stbir__simdf_load2hmerge( out, reg, ptr ) (out) = vcombine_f32( vget_low_f32(reg), vld1_f32( (float const*)(ptr) ) )
1942 // pairwise add of two registers: out = ( a0+a1, a2+a3, b0+b1, b2+b3 ), and the halves version: ( a0+a2, a1+a3, b0+b2, b1+b3 )
1943 #define stbir__simdf_hadd2( out, a, b ) (out) = vcombine_f32( vpadd_f32( vget_low_f32( a ), vget_high_f32( a ) ), vpadd_f32( vget_low_f32( b ), vget_high_f32( b ) ) )
1944 #define stbir__simdf_addhalves2( out, a, b ) (out) = vaddq_f32( vcombine_f32( vget_low_f32( a ), vget_low_f32( b ) ), vcombine_f32( vget_high_f32( a ), vget_high_f32( b ) ) )
1945
1946 #define stbir__simdf_zeroP() vdupq_n_f32(0)
1947 #define stbir__simdf_zero( reg ) (reg) = vdupq_n_f32(0)
1948
1949 #define stbir__simdf_store( ptr, reg ) vst1q_f32( (float*)(ptr), reg )
1950 #define stbir__simdf_store1( ptr, reg ) vst1q_lane_f32( (float*)(ptr), reg, 0)
1951 #define stbir__simdf_store2( ptr, reg ) vst1_f32( (float*)(ptr), vget_low_f32(reg) )
1952 #define stbir__simdf_store2h( ptr, reg ) vst1_f32( (float*)(ptr), vget_high_f32(reg) )
1953
1954 #define stbir__simdi_store( ptr, reg ) vst1q_u32( (uint32_t*)(ptr), reg )
1955 #define stbir__simdi_store2( ptr, reg ) vst1_u32( (uint32_t*)(ptr), vget_low_u32(reg) )
1956
1957 #define stbir__prefetch( ptr )
1958
1959 #define stbir__simdi_expand_u8_to_u32(out0,out1,out2,out3,ireg) \
1960 { \
1961 uint16x8_t l = vmovl_u8( vget_low_u8 ( vreinterpretq_u8_u32(ireg) ) ); \
1962 uint16x8_t h = vmovl_u8( vget_high_u8( vreinterpretq_u8_u32(ireg) ) ); \
1963 out0 = vmovl_u16( vget_low_u16 ( l ) ); \
1964 out1 = vmovl_u16( vget_high_u16( l ) ); \
1965 out2 = vmovl_u16( vget_low_u16 ( h ) ); \
1966 out3 = vmovl_u16( vget_high_u16( h ) ); \
1967 }
1968
1969
1970 #define stbir__simdi_expand_u16_to_u32(out0,out1,ireg) \
1971 { \
1972 uint16x8_t tmp = vreinterpretq_u16_u32(ireg); \
1973 out0 = vmovl_u16( vget_low_u16 ( tmp ) ); \
1974 out1 = vmovl_u16( vget_high_u16( tmp ) ); \
1975 }
1976
1977 #define stbir__simdf_convert_float_to_i32( i, f ) (i) = vreinterpretq_u32_s32( vcvtq_s32_f32(f) )
1978 #define stbir__simdi_to_int( i ) (int)vgetq_lane_u32(i, 0)
1979 #define stbir__simdf_convert_float_to_uint8( f ) ((unsigned char)vgetq_lane_s32(vcvtq_s32_f32(vmaxq_f32(vminq_f32(f,STBIR__CONSTF(STBIR_max_uint8_as_float)),vdupq_n_f32(0))), 0))
1980 #define stbir__simdf_convert_float_to_short( f ) ((unsigned short)vgetq_lane_s32(vcvtq_s32_f32(vmaxq_f32(vminq_f32(f,STBIR__CONSTF(STBIR_max_uint16_as_float)),vdupq_n_f32(0))), 0))
1981 #define stbir__simdi_convert_i32_to_float(out, ireg) (out) = vcvtq_f32_s32( vreinterpretq_s32_u32(ireg) )
1982 #define stbir__simdf_add( out, reg0, reg1 ) (out) = vaddq_f32( reg0, reg1 )
1983 #define stbir__simdf_mult( out, reg0, reg1 ) (out) = vmulq_f32( reg0, reg1 )
1984 #define stbir__simdf_mult_mem( out, reg, ptr ) (out) = vmulq_f32( reg, vld1q_f32( (float const*)(ptr) ) )
1985 #define stbir__simdf_mult1_mem( out, reg, ptr ) (out) = vmulq_f32( reg, vld1q_dup_f32( (float const*)(ptr) ) )
1986 #define stbir__simdf_add_mem( out, reg, ptr ) (out) = vaddq_f32( reg, vld1q_f32( (float const*)(ptr) ) )
1987
1988 #ifdef STBIR_USE_FMA // not on by default to maintain bit identical simd to non-simd (and also x64 no madd to arm madd)
1989 #define stbir__simdf_madd( out, add, mul1, mul2 ) (out) = vfmaq_f32( add, mul1, mul2 )
1990 #define stbir__simdf_madd_mem( out, add, mul, ptr ) (out) = vfmaq_f32( add, mul, vld1q_f32( (float const*)(ptr) ) )
1991 #define stbir__simdf_madd1_mem( out, add, mul, ptr ) (out) = vfmaq_f32( add, mul, vld1q_dup_f32( (float const*)(ptr) ) )
1992 #else
1993 #define stbir__simdf_madd( out, add, mul1, mul2 ) (out) = vaddq_f32( add, vmulq_f32( mul1, mul2 ) )
1994 #define stbir__simdf_madd_mem( out, add, mul, ptr ) (out) = vaddq_f32( add, vmulq_f32( mul, vld1q_f32( (float const*)(ptr) ) ) )
1995 #define stbir__simdf_madd1_mem( out, add, mul, ptr ) (out) = vaddq_f32( add, vmulq_f32( mul, vld1q_dup_f32( (float const*)(ptr) ) ) )
1996 #endif
1997
1998 #define stbir__simdf_add1( out, reg0, reg1 ) (out) = vaddq_f32( reg0, reg1 )
1999
2000 #define stbir__simdf_and( out, reg0, reg1 ) (out) = vreinterpretq_f32_u32( vandq_u32( vreinterpretq_u32_f32(reg0), vreinterpretq_u32_f32(reg1) ) )
2001
2002 // min / max with SSE's nan rule
2003 #if defined( STBIR__ARM64 )
2004 #define stbir__simdf_min( out, reg0, reg1 ) (out) = vminnmq_f32( reg0, reg1 )
2005 #define stbir__simdf_max( out, reg0, reg1 ) (out) = vmaxnmq_f32( reg0, reg1 )
2006 #else
2007 #define stbir__simdf_min( out, reg0, reg1 ) (out) = vbslq_f32( vcltq_f32( reg0, reg1 ), reg0, reg1 )
2008 #define stbir__simdf_max( out, reg0, reg1 ) (out) = vbslq_f32( vcgtq_f32( reg0, reg1 ), reg0, reg1 )
2009 #endif
2010
2011 #define stbir__simdf_0123ABCDto3ABx( out, reg0, reg1 ) (out) = vextq_f32( reg0, reg1, 3 )
2012 #define stbir__simdf_0123ABCDto23Ax( out, reg0, reg1 ) (out) = vextq_f32( reg0, reg1, 2 )
2013
2014 #define stbir__simdf_a1a1( out, alp, ones ) (out) = vzipq_f32(vuzpq_f32(alp, alp).val[1], ones).val[0]
2015 #define stbir__simdf_1a1a( out, alp, ones ) (out) = vzipq_f32(ones, vuzpq_f32(alp, alp).val[0]).val[0]
2016
2017 #if defined( STBIR__ARM64 )
2018
2019 #define stbir__simdf_aaa1( out, alp, ones ) (out) = vcopyq_laneq_f32(vdupq_n_f32(vgetq_lane_f32(alp, 3)), 3, ones, 3)
2020 #define stbir__simdf_1aaa( out, alp, ones ) (out) = vcopyq_laneq_f32(vdupq_n_f32(vgetq_lane_f32(alp, 0)), 0, ones, 0)
2021
2022 #if defined( _MSC_VER ) && !defined(__clang__)
2023 #define stbir_make16(a,b,c,d) vcombine_u8( \
2024 vcreate_u8( (4*a+0) | ((4*a+1)<<8) | ((4*a+2)<<16) | ((4*a+3)<<24) | \
2025 ((stbir_uint64)(4*b+0)<<32) | ((stbir_uint64)(4*b+1)<<40) | ((stbir_uint64)(4*b+2)<<48) | ((stbir_uint64)(4*b+3)<<56)), \
2026 vcreate_u8( (4*c+0) | ((4*c+1)<<8) | ((4*c+2)<<16) | ((4*c+3)<<24) | \
2027 ((stbir_uint64)(4*d+0)<<32) | ((stbir_uint64)(4*d+1)<<40) | ((stbir_uint64)(4*d+2)<<48) | ((stbir_uint64)(4*d+3)<<56) ) )
2028
2029 static stbir__inline uint8x16x2_t stbir_make16x2(float32x4_t rega,float32x4_t regb)
2030 {
2031 uint8x16x2_t r = { vreinterpretq_u8_f32(rega), vreinterpretq_u8_f32(regb) };
2032 return r;
2033 }
2034 #else
2035 #define stbir_make16(a,b,c,d) (uint8x16_t){4*a+0,4*a+1,4*a+2,4*a+3,4*b+0,4*b+1,4*b+2,4*b+3,4*c+0,4*c+1,4*c+2,4*c+3,4*d+0,4*d+1,4*d+2,4*d+3}
2036 #define stbir_make16x2(a,b) (uint8x16x2_t){{vreinterpretq_u8_f32(a),vreinterpretq_u8_f32(b)}}
2037 #endif
2038
2039 #define stbir__simdf_swiz( reg, one, two, three, four ) vreinterpretq_f32_u8( vqtbl1q_u8( vreinterpretq_u8_f32(reg), stbir_make16(one, two, three, four) ) )
2040 #define stbir__simdf_swiz2( rega, regb, one, two, three, four ) vreinterpretq_f32_u8( vqtbl2q_u8( stbir_make16x2(rega,regb), stbir_make16(one, two, three, four) ) )
2041
2042 // an arbitrary byte permute
2043 #define STBIR_SIMD_BYTE_SHUFFLE
2044 #define stbir__simdi_shuffle_bytes( out, reg, table ) (out) = vreinterpretq_u32_u8( vqtbl1q_u8( vreinterpretq_u8_u32( reg ), vreinterpretq_u8_u32( table ) ) )
2045
2046 #define stbir__simdi_16madd( out, reg0, reg1 ) \
2047 { \
2048 int16x8_t r0 = vreinterpretq_s16_u32(reg0); \
2049 int16x8_t r1 = vreinterpretq_s16_u32(reg1); \
2050 int32x4_t tmp0 = vmull_s16( vget_low_s16(r0), vget_low_s16(r1) ); \
2051 int32x4_t tmp1 = vmull_s16( vget_high_s16(r0), vget_high_s16(r1) ); \
2052 (out) = vreinterpretq_u32_s32( vpaddq_s32(tmp0, tmp1) ); \
2053 }
2054
2055 #else
2056
2057 #define stbir__simdf_aaa1( out, alp, ones ) (out) = vsetq_lane_f32(1.0f, vdupq_n_f32(vgetq_lane_f32(alp, 3)), 3)
2058 #define stbir__simdf_1aaa( out, alp, ones ) (out) = vsetq_lane_f32(1.0f, vdupq_n_f32(vgetq_lane_f32(alp, 0)), 0)
2059
2060 #if defined( _MSC_VER ) && !defined(__clang__)
2061 static stbir__inline uint8x8x2_t stbir_make8x2(float32x4_t reg)
2062 {
2063 uint8x8x2_t r = { { vget_low_u8(vreinterpretq_u8_f32(reg)), vget_high_u8(vreinterpretq_u8_f32(reg)) } };
2064 return r;
2065 }
2066 #define stbir_make8(a,b) vcreate_u8( \
2067 (4*a+0) | ((4*a+1)<<8) | ((4*a+2)<<16) | ((4*a+3)<<24) | \
2068 ((stbir_uint64)(4*b+0)<<32) | ((stbir_uint64)(4*b+1)<<40) | ((stbir_uint64)(4*b+2)<<48) | ((stbir_uint64)(4*b+3)<<56) )
2069 #else
2070 #define stbir_make8x2(reg) (uint8x8x2_t){ { vget_low_u8(vreinterpretq_u8_f32(reg)), vget_high_u8(vreinterpretq_u8_f32(reg)) } }
2071 #define stbir_make8(a,b) (uint8x8_t){4*a+0,4*a+1,4*a+2,4*a+3,4*b+0,4*b+1,4*b+2,4*b+3}
2072 #endif
2073
2074 #define stbir__simdf_swiz( reg, one, two, three, four ) vreinterpretq_f32_u8( vcombine_u8( \
2075 vtbl2_u8( stbir_make8x2( reg ), stbir_make8( one, two ) ), \
2076 vtbl2_u8( stbir_make8x2( reg ), stbir_make8( three, four ) ) ) )
2077
2078 #define stbir__simdi_16madd( out, reg0, reg1 ) \
2079 { \
2080 int16x8_t r0 = vreinterpretq_s16_u32(reg0); \
2081 int16x8_t r1 = vreinterpretq_s16_u32(reg1); \
2082 int32x4_t tmp0 = vmull_s16( vget_low_s16(r0), vget_low_s16(r1) ); \
2083 int32x4_t tmp1 = vmull_s16( vget_high_s16(r0), vget_high_s16(r1) ); \
2084 int32x2_t out0 = vpadd_s32( vget_low_s32(tmp0), vget_high_s32(tmp0) ); \
2085 int32x2_t out1 = vpadd_s32( vget_low_s32(tmp1), vget_high_s32(tmp1) ); \
2086 (out) = vreinterpretq_u32_s32( vcombine_s32(out0, out1) ); \
2087 }
2088
2089 #endif
2090
2091 #define stbir__simdi_and( out, reg0, reg1 ) (out) = vandq_u32( reg0, reg1 )
2092 #define stbir__simdi_or( out, reg0, reg1 ) (out) = vorrq_u32( reg0, reg1 )
2093
2094 #define stbir__simdf_pack_to_8bytes(out,aa,bb) \
2095 { \
2096 float32x4_t af = vmaxq_f32( vminq_f32(aa,STBIR__CONSTF(STBIR_max_uint8_as_float) ), vdupq_n_f32(0) ); \
2097 float32x4_t bf = vmaxq_f32( vminq_f32(bb,STBIR__CONSTF(STBIR_max_uint8_as_float) ), vdupq_n_f32(0) ); \
2098 int16x4_t ai = vqmovn_s32( vcvtq_s32_f32( af ) ); \
2099 int16x4_t bi = vqmovn_s32( vcvtq_s32_f32( bf ) ); \
2100 uint8x8_t out8 = vqmovun_s16( vcombine_s16(ai, bi) ); \
2101 out = vreinterpretq_u32_u8( vcombine_u8(out8, out8) ); \
2102 }
2103
2104 #define stbir__simdf_pack_to_8words(out,aa,bb) \
2105 { \
2106 float32x4_t af = vmaxq_f32( vminq_f32(aa,STBIR__CONSTF(STBIR_max_uint16_as_float) ), vdupq_n_f32(0) ); \
2107 float32x4_t bf = vmaxq_f32( vminq_f32(bb,STBIR__CONSTF(STBIR_max_uint16_as_float) ), vdupq_n_f32(0) ); \
2108 int32x4_t ai = vcvtq_s32_f32( af ); \
2109 int32x4_t bi = vcvtq_s32_f32( bf ); \
2110 out = vreinterpretq_u32_u16( vcombine_u16(vqmovun_s32(ai), vqmovun_s32(bi)) ); \
2111 }
2112
2113 #define stbir__interleave_pack_and_store_16_u8( ptr, r0, r1, r2, r3 ) \
2114 { \
2115 int16x4x2_t tmp0 = vzip_s16( vqmovn_s32(vreinterpretq_s32_u32(r0)), vqmovn_s32(vreinterpretq_s32_u32(r2)) ); \
2116 int16x4x2_t tmp1 = vzip_s16( vqmovn_s32(vreinterpretq_s32_u32(r1)), vqmovn_s32(vreinterpretq_s32_u32(r3)) ); \
2117 uint8x8x2_t out = \
2118 { { \
2119 vqmovun_s16( vcombine_s16(tmp0.val[0], tmp0.val[1]) ), \
2120 vqmovun_s16( vcombine_s16(tmp1.val[0], tmp1.val[1]) ), \
2121 } }; \
2122 vst2_u8(ptr, out); \
2123 }
2124
2125 #define stbir__simdf_load4_transposed( o0, o1, o2, o3, ptr ) \
2126 { \
2127 float32x4x4_t tmp = vld4q_f32(ptr); \
2128 o0 = tmp.val[0]; \
2129 o1 = tmp.val[1]; \
2130 o2 = tmp.val[2]; \
2131 o3 = tmp.val[3]; \
2132 }
2133
2134 #define stbir__simdi_32shr( out, reg, imm ) out = vshrq_n_u32( reg, imm )
2135
2136 #if defined( _MSC_VER ) && !defined(__clang__)
2137 #define STBIR__SIMDF_CONST(var, x) __declspec(align(8)) float var[] = { x, x, x, x }
2138 #define STBIR__SIMDI_CONST(var, x) __declspec(align(8)) uint32_t var[] = { x, x, x, x }
2139 #define STBIR__SIMDI_CONST4(var, a, b, c, d) __declspec(align(8)) uint32_t var[] = { a, b, c, d }
2140 #define STBIR__CONSTF(var) (*(const float32x4_t*)var)
2141 #define STBIR__CONSTI(var) (*(const uint32x4_t*)var)
2142 #else
2143 #define STBIR__SIMDF_CONST(var, x) stbir__simdf var = { x, x, x, x }
2144 #define STBIR__SIMDI_CONST(var, x) stbir__simdi var = { x, x, x, x }
2145 #define STBIR__SIMDI_CONST4(var, a, b, c, d) stbir__simdi var = { a, b, c, d }
2146 #define STBIR__CONSTF(var) (var)
2147 #define STBIR__CONSTI(var) (var)
2148 #endif
2149
2150 #define STBIR_SIMD
2151
2152#elif defined(STBIR_WASM)
2153
2154 #include <wasm_simd128.h>
2155
2156 #define stbir__simdf v128_t
2157 #define stbir__simdi v128_t
2158
2159 #define stbir_simdi_castf( reg ) (reg)
2160 #define stbir_simdf_casti( reg ) (reg)
2161
2162 #define stbir__simdf_load( reg, ptr ) (reg) = wasm_v128_load( (void const*)(ptr) )
2163 #define stbir__simdi_load( reg, ptr ) (reg) = wasm_v128_load( (void const*)(ptr) )
2164 #define stbir__simdf_load1( out, ptr ) (out) = wasm_v128_load32_zero( (void const*)(ptr) ) // top values zero, like SSE (a replicated inf would turn into nan against the zero coefficient lanes)
2165 #define stbir__simdf_load1z( out, ptr ) (out) = wasm_v128_load32_zero( (void const*)(ptr) ) // top values must be zero
2166 #define stbir__simdf_frep4( fvar ) wasm_f32x4_splat( fvar )
2167 #define stbir__simdf_load1frep4( out, fvar ) (out) = wasm_f32x4_splat( fvar )
2168 #define stbir__simdf_load2( out, ptr ) (out) = wasm_v128_load64_zero( (void const*)(ptr) ) // top values zero, like SSE
2169 #define stbir__simdf_load2z( out, ptr ) (out) = wasm_v128_load64_zero( (void const*)(ptr) ) // top values must be zero
2170 #define stbir__simdf_load2hmerge( out, reg, ptr ) (out) = wasm_v128_load64_lane( (void const*)(ptr), reg, 1 )
2171// pairwise add of two registers: out = ( a0+a1, a2+a3, b0+b1, b2+b3 ), and the halves version: ( a0+a2, a1+a3, b0+b2, b1+b3 )
2172#define stbir__simdf_hadd2( out, a, b ) (out) = wasm_f32x4_add( wasm_i32x4_shuffle( a, b, 0, 2, 4, 6 ), wasm_i32x4_shuffle( a, b, 1, 3, 5, 7 ) )
2173#define stbir__simdf_addhalves2( out, a, b ) (out) = wasm_f32x4_add( wasm_i32x4_shuffle( a, b, 0, 1, 4, 5 ), wasm_i32x4_shuffle( a, b, 2, 3, 6, 7 ) )
2174
2175 #define stbir__simdf_zeroP() wasm_f32x4_const_splat(0)
2176 #define stbir__simdf_zero( reg ) (reg) = wasm_f32x4_const_splat(0)
2177
2178 #define stbir__simdf_store( ptr, reg ) wasm_v128_store( (void*)(ptr), reg )
2179 #define stbir__simdf_store1( ptr, reg ) wasm_v128_store32_lane( (void*)(ptr), reg, 0 )
2180 #define stbir__simdf_store2( ptr, reg ) wasm_v128_store64_lane( (void*)(ptr), reg, 0 )
2181 #define stbir__simdf_store2h( ptr, reg ) wasm_v128_store64_lane( (void*)(ptr), reg, 1 )
2182
2183 #define stbir__simdi_store( ptr, reg ) wasm_v128_store( (void*)(ptr), reg )
2184 #define stbir__simdi_store2( ptr, reg ) wasm_v128_store64_lane( (void*)(ptr), reg, 0 )
2185
2186 #define stbir__prefetch( ptr )
2187
2188 #define stbir__simdi_expand_u8_to_u32(out0,out1,out2,out3,ireg) \
2189 { \
2190 v128_t l = wasm_u16x8_extend_low_u8x16 ( ireg ); \
2191 v128_t h = wasm_u16x8_extend_high_u8x16( ireg ); \
2192 out0 = wasm_u32x4_extend_low_u16x8 ( l ); \
2193 out1 = wasm_u32x4_extend_high_u16x8( l ); \
2194 out2 = wasm_u32x4_extend_low_u16x8 ( h ); \
2195 out3 = wasm_u32x4_extend_high_u16x8( h ); \
2196 }
2197
2198
2199 #define stbir__simdi_expand_u16_to_u32(out0,out1,ireg) \
2200 { \
2201 out0 = wasm_u32x4_extend_low_u16x8 ( ireg ); \
2202 out1 = wasm_u32x4_extend_high_u16x8( ireg ); \
2203 }
2204
2205 #define stbir__simdf_convert_float_to_i32( i, f ) (i) = wasm_i32x4_trunc_sat_f32x4(f)
2206 #define stbir__simdi_to_int( i ) wasm_i32x4_extract_lane(i, 0)
2207 #define stbir__simdf_convert_float_to_uint8( f ) ((unsigned char)wasm_i32x4_extract_lane(wasm_i32x4_trunc_sat_f32x4(wasm_f32x4_max(wasm_f32x4_min(f,STBIR_max_uint8_as_float),wasm_f32x4_const_splat(0))), 0))
2208 #define stbir__simdf_convert_float_to_short( f ) ((unsigned short)wasm_i32x4_extract_lane(wasm_i32x4_trunc_sat_f32x4(wasm_f32x4_max(wasm_f32x4_min(f,STBIR_max_uint16_as_float),wasm_f32x4_const_splat(0))), 0))
2209 #define stbir__simdi_convert_i32_to_float(out, ireg) (out) = wasm_f32x4_convert_i32x4(ireg)
2210 #define stbir__simdf_add( out, reg0, reg1 ) (out) = wasm_f32x4_add( reg0, reg1 )
2211 #define stbir__simdf_mult( out, reg0, reg1 ) (out) = wasm_f32x4_mul( reg0, reg1 )
2212 #define stbir__simdf_mult_mem( out, reg, ptr ) (out) = wasm_f32x4_mul( reg, wasm_v128_load( (void const*)(ptr) ) )
2213 #define stbir__simdf_mult1_mem( out, reg, ptr ) (out) = wasm_f32x4_mul( reg, wasm_v128_load32_splat( (void const*)(ptr) ) )
2214 #define stbir__simdf_add_mem( out, reg, ptr ) (out) = wasm_f32x4_add( reg, wasm_v128_load( (void const*)(ptr) ) )
2215
2216 #define stbir__simdf_madd( out, add, mul1, mul2 ) (out) = wasm_f32x4_add( add, wasm_f32x4_mul( mul1, mul2 ) )
2217 #define stbir__simdf_madd_mem( out, add, mul, ptr ) (out) = wasm_f32x4_add( add, wasm_f32x4_mul( mul, wasm_v128_load( (void const*)(ptr) ) ) )
2218 #define stbir__simdf_madd1_mem( out, add, mul, ptr ) (out) = wasm_f32x4_add( add, wasm_f32x4_mul( mul, wasm_v128_load32_splat( (void const*)(ptr) ) ) )
2219
2220 #define stbir__simdf_add1( out, reg0, reg1 ) (out) = wasm_f32x4_add( reg0, reg1 )
2221
2222 #define stbir__simdf_and( out, reg0, reg1 ) (out) = wasm_v128_and( reg0, reg1 )
2223
2224 #define stbir__simdf_min( out, reg0, reg1 ) (out) = wasm_f32x4_min( reg0, reg1 )
2225 #define stbir__simdf_max( out, reg0, reg1 ) (out) = wasm_f32x4_max( reg0, reg1 )
2226
2227 #define stbir__simdf_0123ABCDto3ABx( out, reg0, reg1 ) (out) = wasm_i32x4_shuffle( reg0, reg1, 3, 4, 5, -1 )
2228 #define stbir__simdf_0123ABCDto23Ax( out, reg0, reg1 ) (out) = wasm_i32x4_shuffle( reg0, reg1, 2, 3, 4, -1 )
2229
2230 #define stbir__simdf_aaa1(out,alp,ones) (out) = wasm_i32x4_shuffle(alp, ones, 3, 3, 3, 4)
2231 #define stbir__simdf_1aaa(out,alp,ones) (out) = wasm_i32x4_shuffle(alp, ones, 4, 0, 0, 0)
2232 #define stbir__simdf_a1a1(out,alp,ones) (out) = wasm_i32x4_shuffle(alp, ones, 1, 4, 3, 4)
2233 #define stbir__simdf_1a1a(out,alp,ones) (out) = wasm_i32x4_shuffle(alp, ones, 4, 0, 4, 2)
2234
2235 #define stbir__simdf_swiz( reg, one, two, three, four ) wasm_i32x4_shuffle(reg, reg, one, two, three, four)
2236
2237 // an arbitrary byte permute
2238 #define STBIR_SIMD_BYTE_SHUFFLE
2239 #define stbir__simdi_shuffle_bytes( out, reg, table ) (out) = wasm_i8x16_swizzle( reg, table )
2240
2241 #define stbir__simdi_and( out, reg0, reg1 ) (out) = wasm_v128_and( reg0, reg1 )
2242 #define stbir__simdi_or( out, reg0, reg1 ) (out) = wasm_v128_or( reg0, reg1 )
2243 #define stbir__simdi_16madd( out, reg0, reg1 ) (out) = wasm_i32x4_dot_i16x8( reg0, reg1 )
2244
2245 #define stbir__simdf_pack_to_8bytes(out,aa,bb) \
2246 { \
2247 v128_t af = wasm_f32x4_max( wasm_f32x4_min(aa, STBIR_max_uint8_as_float), wasm_f32x4_const_splat(0) ); \
2248 v128_t bf = wasm_f32x4_max( wasm_f32x4_min(bb, STBIR_max_uint8_as_float), wasm_f32x4_const_splat(0) ); \
2249 v128_t ai = wasm_i32x4_trunc_sat_f32x4( af ); \
2250 v128_t bi = wasm_i32x4_trunc_sat_f32x4( bf ); \
2251 v128_t out16 = wasm_i16x8_narrow_i32x4( ai, bi ); \
2252 out = wasm_u8x16_narrow_i16x8( out16, out16 ); \
2253 }
2254
2255 #define stbir__simdf_pack_to_8words(out,aa,bb) \
2256 { \
2257 v128_t af = wasm_f32x4_max( wasm_f32x4_min(aa, STBIR_max_uint16_as_float), wasm_f32x4_const_splat(0)); \
2258 v128_t bf = wasm_f32x4_max( wasm_f32x4_min(bb, STBIR_max_uint16_as_float), wasm_f32x4_const_splat(0)); \
2259 v128_t ai = wasm_i32x4_trunc_sat_f32x4( af ); \
2260 v128_t bi = wasm_i32x4_trunc_sat_f32x4( bf ); \
2261 out = wasm_u16x8_narrow_i32x4( ai, bi ); \
2262 }
2263
2264 #define stbir__interleave_pack_and_store_16_u8( ptr, r0, r1, r2, r3 ) \
2265 { \
2266 v128_t tmp0 = wasm_i16x8_narrow_i32x4(r0, r1); \
2267 v128_t tmp1 = wasm_i16x8_narrow_i32x4(r2, r3); \
2268 v128_t tmp = wasm_u8x16_narrow_i16x8(tmp0, tmp1); \
2269 tmp = wasm_i8x16_shuffle(tmp, tmp, 0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15); \
2270 wasm_v128_store( (void*)(ptr), tmp); \
2271 }
2272
2273 #define stbir__simdf_load4_transposed( o0, o1, o2, o3, ptr ) \
2274 { \
2275 v128_t t0 = wasm_v128_load( ptr ); \
2276 v128_t t1 = wasm_v128_load( ptr+4 ); \
2277 v128_t t2 = wasm_v128_load( ptr+8 ); \
2278 v128_t t3 = wasm_v128_load( ptr+12 ); \
2279 v128_t s0 = wasm_i32x4_shuffle(t0, t1, 0, 4, 2, 6); \
2280 v128_t s1 = wasm_i32x4_shuffle(t0, t1, 1, 5, 3, 7); \
2281 v128_t s2 = wasm_i32x4_shuffle(t2, t3, 0, 4, 2, 6); \
2282 v128_t s3 = wasm_i32x4_shuffle(t2, t3, 1, 5, 3, 7); \
2283 o0 = wasm_i32x4_shuffle(s0, s2, 0, 1, 4, 5); \
2284 o1 = wasm_i32x4_shuffle(s1, s3, 0, 1, 4, 5); \
2285 o2 = wasm_i32x4_shuffle(s0, s2, 2, 3, 6, 7); \
2286 o3 = wasm_i32x4_shuffle(s1, s3, 2, 3, 6, 7); \
2287 }
2288
2289 #define stbir__simdi_32shr( out, reg, imm ) out = wasm_u32x4_shr( reg, imm )
2290
2291 typedef float stbir__f32x4 __attribute__((__vector_size__(16), __aligned__(16)));
2292 #define STBIR__SIMDF_CONST(var, x) stbir__simdf var = (v128_t)(stbir__f32x4){ x, x, x, x }
2293 #define STBIR__SIMDI_CONST(var, x) stbir__simdi var = { x, x, x, x }
2294 #define STBIR__SIMDI_CONST4(var, a, b, c, d) stbir__simdi var = { a, b, c, d }
2295 #define STBIR__CONSTF(var) (var)
2296 #define STBIR__CONSTI(var) (var)
2297
2298 #define STBIR_SIMD
2299
2300#endif // SSE2/NEON/WASM
2301
2302#endif // NO SIMD
2303
2304// the X-wide names
2305#if defined(STBIR_SIMD16)
2306 #define STBIR__U16_DECODE_BLOCK 16 // words per block in the u16 decoders (one 32 byte load)
2307 #define stbir__simdfX stbir__simdf16
2308 #define stbir__simdiX stbir__simdi16
2309 #define stbir__simdfX_load stbir__simdf16_load
2310 #define stbir__simdiX_load stbir__simdi16_load
2311 #define stbir__simdfX_mult stbir__simdf16_mult
2312 #define stbir__simdfX_add_mem stbir__simdf16_add_mem
2313 #define stbir__simdfX_madd_mem stbir__simdf16_madd_mem
2314 #define stbir__simdfX_store stbir__simdf16_store
2315 #define stbir__simdiX_store stbir__simdi16_store
2316 #define stbir__simdf_frepX stbir__simdf16_frep16
2317 #define stbir__simdfX_madd stbir__simdf16_madd
2318 #define stbir__simdfX_min stbir__simdf16_min
2319 #define stbir__simdfX_max stbir__simdf16_max
2320 #define stbir__simdfX_aaa1 stbir__simdf16_aaa1
2321 #define stbir__simdfX_1aaa stbir__simdf16_1aaa
2322 #define stbir__simdfX_a1a1 stbir__simdf16_a1a1
2323 #define stbir__simdfX_1a1a stbir__simdf16_1a1a
2324 #define stbir__simdfX_convert_float_to_i32 stbir__simdf16_convert_float_to_i32
2325 #define stbir__simdfX_pack_to_words stbir__simdf16_pack_to_32words
2326 #define stbir__simdfX_zero stbir__simdf16_zero
2327 #define STBIR_onesX STBIR_ones16
2328 #define STBIR_max_uint8_as_floatX STBIR_max_uint8_as_float16
2329 #define STBIR_max_uint16_as_floatX STBIR_max_uint16_as_float16
2330 #define STBIR_simd_point5X STBIR_simd_point516
2331 #define stbir__simdfX_float_count 16
2332 #define stbir__simdfX_cast_to_simdf4( val ) stbir__simdf16_cast_to_simdf4( val )
2333 #define stbir__simdfX_0123to1230 stbir__simdf16_0123to1230
2334 #define stbir__simdfX_0123to2103 stbir__simdf16_0123to2103
2335#elif defined(STBIR_SIMD8)
2336 #define STBIR__U16_DECODE_BLOCK 8
2337 #define stbir__simdfX stbir__simdf8
2338 #define stbir__simdiX stbir__simdi8
2339 #define stbir__simdfX_load stbir__simdf8_load
2340 #define stbir__simdiX_load stbir__simdi8_load
2341 #define stbir__simdfX_mult stbir__simdf8_mult
2342 #define stbir__simdfX_add_mem stbir__simdf8_add_mem
2343 #define stbir__simdfX_madd_mem stbir__simdf8_madd_mem
2344 #define stbir__simdfX_store stbir__simdf8_store
2345 #define stbir__simdiX_store stbir__simdi8_store
2346 #define stbir__simdf_frepX stbir__simdf8_frep8
2347 #define stbir__simdfX_madd stbir__simdf8_madd
2348 #define stbir__simdfX_min stbir__simdf8_min
2349 #define stbir__simdfX_max stbir__simdf8_max
2350 #define stbir__simdfX_aaa1 stbir__simdf8_aaa1
2351 #define stbir__simdfX_1aaa stbir__simdf8_1aaa
2352 #define stbir__simdfX_a1a1 stbir__simdf8_a1a1
2353 #define stbir__simdfX_1a1a stbir__simdf8_1a1a
2354 #define stbir__simdfX_convert_float_to_i32 stbir__simdf8_convert_float_to_i32
2355 #define stbir__simdfX_pack_to_words stbir__simdf8_pack_to_16words
2356 #define stbir__simdfX_zero stbir__simdf8_zero
2357 #define STBIR_onesX STBIR_ones8
2358 #define STBIR_max_uint8_as_floatX STBIR_max_uint8_as_float8
2359 #define STBIR_max_uint16_as_floatX STBIR_max_uint16_as_float8
2360 #define STBIR_simd_point5X STBIR_simd_point58
2361 #define stbir__simdfX_float_count 8
2362 #define stbir__simdfX_cast_to_simdf4( val ) stbir__if_simdf8_cast_to_simdf4( val )
2363 #define stbir__simdfX_0123to1230 stbir__simdf8_0123to12301230
2364 #define stbir__simdfX_0123to2103 stbir__simdf8_0123to21032103
2365#else
2366 #define STBIR__U16_DECODE_BLOCK 8
2367 #define stbir__simdfX stbir__simdf
2368 #define stbir__simdiX stbir__simdi
2369 #define stbir__simdfX_load stbir__simdf_load
2370 #define stbir__simdiX_load stbir__simdi_load
2371 #define stbir__simdfX_mult stbir__simdf_mult
2372 #define stbir__simdfX_add_mem stbir__simdf_add_mem
2373 #define stbir__simdfX_madd_mem stbir__simdf_madd_mem
2374 #define stbir__simdfX_store stbir__simdf_store
2375 #define stbir__simdiX_store stbir__simdi_store
2376 #define stbir__simdf_frepX stbir__simdf_frep4
2377 #define stbir__simdfX_madd stbir__simdf_madd
2378 #define stbir__simdfX_min stbir__simdf_min
2379 #define stbir__simdfX_max stbir__simdf_max
2380 #define stbir__simdfX_aaa1 stbir__simdf_aaa1
2381 #define stbir__simdfX_1aaa stbir__simdf_1aaa
2382 #define stbir__simdfX_a1a1 stbir__simdf_a1a1
2383 #define stbir__simdfX_1a1a stbir__simdf_1a1a
2384 #define stbir__simdfX_convert_float_to_i32 stbir__simdf_convert_float_to_i32
2385 #define stbir__simdfX_pack_to_words stbir__simdf_pack_to_8words
2386 #define stbir__simdfX_zero stbir__simdf_zero
2387 #define STBIR_onesX STBIR__CONSTF(STBIR_ones)
2388 #define STBIR_simd_point5X STBIR__CONSTF(STBIR_simd_point5)
2389 #define STBIR_max_uint8_as_floatX STBIR__CONSTF(STBIR_max_uint8_as_float)
2390 #define STBIR_max_uint16_as_floatX STBIR__CONSTF(STBIR_max_uint16_as_float)
2391 #define stbir__simdfX_float_count 4
2392 #define stbir__simdfX_cast_to_simdf4( val ) ( val )
2393 #define stbir__if_simdf8_cast_to_simdf4( val ) ( val )
2394 #define stbir__simdfX_0123to1230 stbir__simdf_0123to1230
2395 #define stbir__simdfX_0123to2103 stbir__simdf_0123to2103
2396#endif
2397
2398
2399#if defined(STBIR_NEON) && !defined(_M_ARM) && !defined(__arm__)
2400
2401 #if defined( _MSC_VER ) && !defined(__clang__)
2402 typedef __int16 stbir__FP16;
2403 #else
2404 typedef float16_t stbir__FP16;
2405 #endif
2406
2407#else // no NEON, or 32-bit ARM for MSVC
2408
2409 typedef union stbir__FP16
2410 {
2411 unsigned short u;
2412 } stbir__FP16;
2413
2414#endif
2415
2416#if (!defined(STBIR_NEON) && !defined(STBIR_FP16C)) || (defined(STBIR_NEON) && defined(_M_ARM)) || (defined(STBIR_NEON) && defined(__arm__))
2417
2418 // Fabian's half float routines, see: https://gist.github.com/rygorous/2156668
2419
2420 static stbir__inline float stbir__half_to_float( stbir__FP16 h )
2421 {
2422 static const stbir__FP32 magic = { (254 - 15) << 23 };
2423 static const stbir__FP32 was_infnan = { (127 + 16) << 23 };
2424 stbir__FP32 o;
2425
2426 o.u = (h.u & 0x7fff) << 13; // exponent/mantissa bits
2427 o.f *= magic.f; // exponent adjust
2428 if (o.f >= was_infnan.f) // make sure Inf/NaN survive
2429 o.u |= 255 << 23;
2430 o.u |= (h.u & 0x8000) << 16; // sign bit
2431 return o.f;
2432 }
2433
2434 static stbir__inline stbir__FP16 stbir__float_to_half(float val)
2435 {
2436 stbir__FP32 f32infty = { 255 << 23 };
2437 stbir__FP32 f16max = { (127 + 16) << 23 };
2438 stbir__FP32 denorm_magic = { ((127 - 15) + (23 - 10) + 1) << 23 };
2439 unsigned int sign_mask = 0x80000000u;
2440 stbir__FP16 o = { 0 };
2441 stbir__FP32 f;
2442 unsigned int sign;
2443
2444 f.f = val;
2445 sign = f.u & sign_mask;
2446 f.u ^= sign;
2447
2448 if (f.u >= f16max.u) // result is Inf or NaN (all exponent bits set)
2449 o.u = (f.u > f32infty.u) ? 0x7e00 : 0x7c00; // NaN->qNaN and Inf->Inf
2450 else // (De)normalized number or zero
2451 {
2452 if (f.u < (113 << 23)) // resulting FP16 is subnormal or zero
2453 {
2454 // use a magic value to align our 10 mantissa bits at the bottom of
2455 // the float. as long as FP addition is round-to-nearest-even this
2456 // just works.
2457 f.f += denorm_magic.f;
2458 // and one integer subtract of the bias later, we have our final float!
2459 o.u = (unsigned short) ( f.u - denorm_magic.u );
2460 }
2461 else
2462 {
2463 unsigned int mant_odd = (f.u >> 13) & 1; // resulting mantissa is odd
2464 // update exponent, rounding bias part 1
2465 f.u = f.u + ((15u - 127) << 23) + 0xfff;
2466 // rounding bias part 2
2467 f.u += mant_odd;
2468 // take the bits!
2469 o.u = (unsigned short) ( f.u >> 13 );
2470 }
2471 }
2472
2473 o.u |= sign >> 16;
2474 return o;
2475 }
2476
2477#endif
2478
2479
2480#if defined(STBIR_FP16C)
2481
2482 #include <immintrin.h>
2483
2484 static stbir__inline void stbir__half_to_float_SIMD(float * output, stbir__FP16 const * input)
2485 {
2486 _mm256_storeu_ps( (float*)output, _mm256_cvtph_ps( _mm_loadu_si128( (__m128i const* )input ) ) );
2487 }
2488
2489 static stbir__inline void stbir__float_to_half_SIMD(stbir__FP16 * output, float const * input)
2490 {
2491 _mm_storeu_si128( (__m128i*)output, _mm256_cvtps_ph( _mm256_loadu_ps( input ), 0 ) );
2492 }
2493
2494 static stbir__inline float stbir__half_to_float( stbir__FP16 h )
2495 {
2496 return _mm_cvtss_f32( _mm_cvtph_ps( _mm_cvtsi32_si128( (int)h.u ) ) );
2497 }
2498
2499 static stbir__inline stbir__FP16 stbir__float_to_half( float f )
2500 {
2501 stbir__FP16 h;
2502 h.u = (unsigned short) _mm_cvtsi128_si32( _mm_cvtps_ph( _mm_set_ss( f ), 0 ) );
2503 return h;
2504 }
2505
2506#elif defined(STBIR_SSE2)
2507
2508 // Fabian's half float routines, see: https://gist.github.com/rygorous/2156668
2509 stbir__inline static void stbir__half_to_float_SIMD(float * output, void const * input)
2510 {
2511 static const STBIR__SIMDI_CONST(mask_nosign, 0x7fff);
2512 static const STBIR__SIMDI_CONST(smallest_normal, 0x0400);
2513 static const STBIR__SIMDI_CONST(infinity, 0x7c00);
2514 static const STBIR__SIMDI_CONST(expadjust_normal, (127 - 15) << 23);
2515 static const STBIR__SIMDI_CONST(magic_denorm, 113 << 23);
2516
2517 __m128i i = _mm_loadu_si128 ( (__m128i const*)(input) );
2518 __m128i h = _mm_unpacklo_epi16 ( i, _mm_setzero_si128() );
2519 __m128i mnosign = STBIR__CONSTI(mask_nosign);
2520 __m128i eadjust = STBIR__CONSTI(expadjust_normal);
2521 __m128i smallest = STBIR__CONSTI(smallest_normal);
2522 __m128i infty = STBIR__CONSTI(infinity);
2523 __m128i expmant = _mm_and_si128(mnosign, h);
2524 __m128i justsign = _mm_xor_si128(h, expmant);
2525 __m128i b_notinfnan = _mm_cmpgt_epi32(infty, expmant);
2526 __m128i b_isdenorm = _mm_cmpgt_epi32(smallest, expmant);
2527 __m128i shifted = _mm_slli_epi32(expmant, 13);
2528 __m128i adj_infnan = _mm_andnot_si128(b_notinfnan, eadjust);
2529 __m128i adjusted = _mm_add_epi32(eadjust, shifted);
2530 __m128i den1 = _mm_add_epi32(shifted, STBIR__CONSTI(magic_denorm));
2531 __m128i adjusted2 = _mm_add_epi32(adjusted, adj_infnan);
2532 __m128 den2 = _mm_sub_ps(_mm_castsi128_ps(den1), *(const __m128 *)&magic_denorm);
2533 __m128 adjusted3 = _mm_and_ps(den2, _mm_castsi128_ps(b_isdenorm));
2534 __m128 adjusted4 = _mm_andnot_ps(_mm_castsi128_ps(b_isdenorm), _mm_castsi128_ps(adjusted2));
2535 __m128 adjusted5 = _mm_or_ps(adjusted3, adjusted4);
2536 __m128i sign = _mm_slli_epi32(justsign, 16);
2537 __m128 final = _mm_or_ps(adjusted5, _mm_castsi128_ps(sign));
2538 stbir__simdf_store( output + 0, final );
2539
2540 h = _mm_unpackhi_epi16 ( i, _mm_setzero_si128() );
2541 expmant = _mm_and_si128(mnosign, h);
2542 justsign = _mm_xor_si128(h, expmant);
2543 b_notinfnan = _mm_cmpgt_epi32(infty, expmant);
2544 b_isdenorm = _mm_cmpgt_epi32(smallest, expmant);
2545 shifted = _mm_slli_epi32(expmant, 13);
2546 adj_infnan = _mm_andnot_si128(b_notinfnan, eadjust);
2547 adjusted = _mm_add_epi32(eadjust, shifted);
2548 den1 = _mm_add_epi32(shifted, STBIR__CONSTI(magic_denorm));
2549 adjusted2 = _mm_add_epi32(adjusted, adj_infnan);
2550 den2 = _mm_sub_ps(_mm_castsi128_ps(den1), *(const __m128 *)&magic_denorm);
2551 adjusted3 = _mm_and_ps(den2, _mm_castsi128_ps(b_isdenorm));
2552 adjusted4 = _mm_andnot_ps(_mm_castsi128_ps(b_isdenorm), _mm_castsi128_ps(adjusted2));
2553 adjusted5 = _mm_or_ps(adjusted3, adjusted4);
2554 sign = _mm_slli_epi32(justsign, 16);
2555 final = _mm_or_ps(adjusted5, _mm_castsi128_ps(sign));
2556 stbir__simdf_store( output + 4, final );
2557
2558 // ~38 SSE2 ops for 8 values
2559 }
2560
2561 // Fabian's round-to-nearest-even float to half
2562 // ~48 SSE2 ops for 8 output
2563 stbir__inline static void stbir__float_to_half_SIMD(void * output, float const * input)
2564 {
2565 static const STBIR__SIMDI_CONST(mask_sign, 0x80000000u);
2566 static const STBIR__SIMDI_CONST(c_f16max, (127 + 16) << 23); // all FP32 values >=this round to +inf
2567 static const STBIR__SIMDI_CONST(c_nanbit, 0x200);
2568 static const STBIR__SIMDI_CONST(c_infty_as_fp16, 0x7c00);
2569 static const STBIR__SIMDI_CONST(c_min_normal, (127 - 14) << 23); // smallest FP32 that yields a normalized FP16
2570 static const STBIR__SIMDI_CONST(c_subnorm_magic, ((127 - 15) + (23 - 10) + 1) << 23);
2571 static const STBIR__SIMDI_CONST(c_normal_bias, 0xfff - ((127 - 15) << 23)); // adjust exponent and add mantissa rounding
2572
2573 __m128 f = _mm_loadu_ps(input);
2574 __m128 msign = _mm_castsi128_ps(STBIR__CONSTI(mask_sign));
2575 __m128 justsign = _mm_and_ps(msign, f);
2576 __m128 absf = _mm_xor_ps(f, justsign);
2577 __m128i absf_int = _mm_castps_si128(absf); // the cast is "free" (extra bypass latency, but no thruput hit)
2578 __m128i f16max = STBIR__CONSTI(c_f16max);
2579 __m128 b_isnan = _mm_cmpunord_ps(absf, absf); // is this a NaN?
2580 __m128i b_isregular = _mm_cmpgt_epi32(f16max, absf_int); // (sub)normalized or special?
2581 __m128i nanbit = _mm_and_si128(_mm_castps_si128(b_isnan), STBIR__CONSTI(c_nanbit));
2582 __m128i inf_or_nan = _mm_or_si128(nanbit, STBIR__CONSTI(c_infty_as_fp16)); // output for specials
2583
2584 __m128i min_normal = STBIR__CONSTI(c_min_normal);
2585 __m128i b_issub = _mm_cmpgt_epi32(min_normal, absf_int);
2586
2587 // "result is subnormal" path
2588 __m128 subnorm1 = _mm_add_ps(absf, _mm_castsi128_ps(STBIR__CONSTI(c_subnorm_magic))); // magic value to round output mantissa
2589 __m128i subnorm2 = _mm_sub_epi32(_mm_castps_si128(subnorm1), STBIR__CONSTI(c_subnorm_magic)); // subtract out bias
2590
2591 // "result is normal" path
2592 __m128i mantoddbit = _mm_slli_epi32(absf_int, 31 - 13); // shift bit 13 (mantissa LSB) to sign
2593 __m128i mantodd = _mm_srai_epi32(mantoddbit, 31); // -1 if FP16 mantissa odd, else 0
2594
2595 __m128i round1 = _mm_add_epi32(absf_int, STBIR__CONSTI(c_normal_bias));
2596 __m128i round2 = _mm_sub_epi32(round1, mantodd); // if mantissa LSB odd, bias towards rounding up (RTNE)
2597 __m128i normal = _mm_srli_epi32(round2, 13); // rounded result
2598
2599 // combine the two non-specials
2600 __m128i nonspecial = _mm_or_si128(_mm_and_si128(subnorm2, b_issub), _mm_andnot_si128(b_issub, normal));
2601
2602 // merge in specials as well
2603 __m128i joined = _mm_or_si128(_mm_and_si128(nonspecial, b_isregular), _mm_andnot_si128(b_isregular, inf_or_nan));
2604
2605 __m128i sign_shift = _mm_srai_epi32(_mm_castps_si128(justsign), 16);
2606 __m128i final2, final= _mm_or_si128(joined, sign_shift);
2607
2608 f = _mm_loadu_ps(input+4);
2609 justsign = _mm_and_ps(msign, f);
2610 absf = _mm_xor_ps(f, justsign);
2611 absf_int = _mm_castps_si128(absf); // the cast is "free" (extra bypass latency, but no thruput hit)
2612 b_isnan = _mm_cmpunord_ps(absf, absf); // is this a NaN?
2613 b_isregular = _mm_cmpgt_epi32(f16max, absf_int); // (sub)normalized or special?
2614 nanbit = _mm_and_si128(_mm_castps_si128(b_isnan), STBIR__CONSTI(c_nanbit));
2615 inf_or_nan = _mm_or_si128(nanbit, STBIR__CONSTI(c_infty_as_fp16)); // output for specials
2616
2617 b_issub = _mm_cmpgt_epi32(min_normal, absf_int);
2618
2619 // "result is subnormal" path
2620 subnorm1 = _mm_add_ps(absf, _mm_castsi128_ps(STBIR__CONSTI(c_subnorm_magic))); // magic value to round output mantissa
2621 subnorm2 = _mm_sub_epi32(_mm_castps_si128(subnorm1), STBIR__CONSTI(c_subnorm_magic)); // subtract out bias
2622
2623 // "result is normal" path
2624 mantoddbit = _mm_slli_epi32(absf_int, 31 - 13); // shift bit 13 (mantissa LSB) to sign
2625 mantodd = _mm_srai_epi32(mantoddbit, 31); // -1 if FP16 mantissa odd, else 0
2626
2627 round1 = _mm_add_epi32(absf_int, STBIR__CONSTI(c_normal_bias));
2628 round2 = _mm_sub_epi32(round1, mantodd); // if mantissa LSB odd, bias towards rounding up (RTNE)
2629 normal = _mm_srli_epi32(round2, 13); // rounded result
2630
2631 // combine the two non-specials
2632 nonspecial = _mm_or_si128(_mm_and_si128(subnorm2, b_issub), _mm_andnot_si128(b_issub, normal));
2633
2634 // merge in specials as well
2635 joined = _mm_or_si128(_mm_and_si128(nonspecial, b_isregular), _mm_andnot_si128(b_isregular, inf_or_nan));
2636
2637 sign_shift = _mm_srai_epi32(_mm_castps_si128(justsign), 16);
2638 final2 = _mm_or_si128(joined, sign_shift);
2639 final = _mm_packs_epi32(final, final2);
2640 stbir__simdi_store( output,final );
2641 }
2642
2643#elif defined(STBIR_NEON) && defined(_MSC_VER) && defined(STBIR__ARM64) && !defined(__clang__) // 64-bit ARM on MSVC (not clang)
2644
2645 static stbir__inline void stbir__half_to_float_SIMD(float * output, stbir__FP16 const * input)
2646 {
2647 float16x4_t in0 = vld1_f16(input + 0);
2648 float16x4_t in1 = vld1_f16(input + 4);
2649 vst1q_f32(output + 0, vcvt_f32_f16(in0));
2650 vst1q_f32(output + 4, vcvt_f32_f16(in1));
2651 }
2652
2653 static stbir__inline void stbir__float_to_half_SIMD(stbir__FP16 * output, float const * input)
2654 {
2655 float16x4_t out0 = vcvt_f16_f32(vld1q_f32(input + 0));
2656 float16x4_t out1 = vcvt_f16_f32(vld1q_f32(input + 4));
2657 vst1_f16(output+0, out0);
2658 vst1_f16(output+4, out1);
2659 }
2660
2661 static stbir__inline float stbir__half_to_float( stbir__FP16 h )
2662 {
2663 return vgetq_lane_f32(vcvt_f32_f16(vld1_dup_f16(&h)), 0);
2664 }
2665
2666 static stbir__inline stbir__FP16 stbir__float_to_half( float f )
2667 {
2668 return vget_lane_f16(vcvt_f16_f32(vdupq_n_f32(f)), 0).n16_u16[0];
2669 }
2670
2671#elif defined(STBIR_NEON) && defined( STBIR__ARM64 ) // 64-bit ARM
2672
2673 static stbir__inline void stbir__half_to_float_SIMD(float * output, stbir__FP16 const * input)
2674 {
2675 float16x8_t in = vld1q_f16(input);
2676 vst1q_f32(output + 0, vcvt_f32_f16(vget_low_f16(in)));
2677 vst1q_f32(output + 4, vcvt_f32_f16(vget_high_f16(in)));
2678 }
2679
2680 static stbir__inline void stbir__float_to_half_SIMD(stbir__FP16 * output, float const * input)
2681 {
2682 float16x4_t out0 = vcvt_f16_f32(vld1q_f32(input + 0));
2683 float16x4_t out1 = vcvt_f16_f32(vld1q_f32(input + 4));
2684 vst1q_f16(output, vcombine_f16(out0, out1));
2685 }
2686
2687 static stbir__inline float stbir__half_to_float( stbir__FP16 h )
2688 {
2689 return vgetq_lane_f32(vcvt_f32_f16(vdup_n_f16(h)), 0);
2690 }
2691
2692 static stbir__inline stbir__FP16 stbir__float_to_half( float f )
2693 {
2694 return vget_lane_f16(vcvt_f16_f32(vdupq_n_f32(f)), 0);
2695 }
2696
2697#elif defined(STBIR_WASM) || (defined(STBIR_NEON) && (defined(_MSC_VER) || defined(_M_ARM) || defined(__arm__))) // WASM or 32-bit ARM on MSVC/clang
2698
2699 static stbir__inline void stbir__half_to_float_SIMD(float * output, stbir__FP16 const * input)
2700 {
2701 int i;
2702 for (i=0; i<8; i++)
2703 {
2704 output[i] = stbir__half_to_float(input[i]);
2705 }
2706 }
2707 static stbir__inline void stbir__float_to_half_SIMD(stbir__FP16 * output, float const * input)
2708 {
2709 int i;
2710 for (i=0; i<8; i++)
2711 {
2712 output[i] = stbir__float_to_half(input[i]);
2713 }
2714 }
2715
2716#endif
2717
2718
2719#ifdef STBIR_SIMD
2720
2721#define stbir__simdf_0123to3333( out, reg ) (out) = stbir__simdf_swiz( reg, 3,3,3,3 )
2722#define stbir__simdf_0123to2222( out, reg ) (out) = stbir__simdf_swiz( reg, 2,2,2,2 )
2723#define stbir__simdf_0123to1111( out, reg ) (out) = stbir__simdf_swiz( reg, 1,1,1,1 )
2724#define stbir__simdf_0123to0000( out, reg ) (out) = stbir__simdf_swiz( reg, 0,0,0,0 )
2725#define stbir__simdf_0123to0001( out, reg ) (out) = stbir__simdf_swiz( reg, 0,0,0,1 )
2726#define stbir__simdf_0123to1122( out, reg ) (out) = stbir__simdf_swiz( reg, 1,1,2,2 )
2727#define stbir__simdf_0123to2333( out, reg ) (out) = stbir__simdf_swiz( reg, 2,3,3,3 )
2728#define stbir__simdf_0123to1230( out, reg ) (out) = stbir__simdf_swiz( reg, 1,2,3,0 )
2729#define stbir__simdf_0123to2103( out, reg ) (out) = stbir__simdf_swiz( reg, 2,1,0,3 )
2730#define stbir__simdf_0123to3210( out, reg ) (out) = stbir__simdf_swiz( reg, 3,2,1,0 )
2731#define stbir__simdf_0123to2301( out, reg ) (out) = stbir__simdf_swiz( reg, 2,3,0,1 )
2732#define stbir__simdf_0123to3012( out, reg ) (out) = stbir__simdf_swiz( reg, 3,0,1,2 )
2733#define stbir__simdf_0123to0011( out, reg ) (out) = stbir__simdf_swiz( reg, 0,0,1,1 )
2734#define stbir__simdf_0123to2233( out, reg ) (out) = stbir__simdf_swiz( reg, 2,2,3,3 )
2735#define stbir__simdf_0123to1133( out, reg ) (out) = stbir__simdf_swiz( reg, 1,1,3,3 )
2736#define stbir__simdf_0123to0022( out, reg ) (out) = stbir__simdf_swiz( reg, 0,0,2,2 )
2737#define stbir__simdf_0123to1032( out, reg ) (out) = stbir__simdf_swiz( reg, 1,0,3,2 )
2738
2739typedef union stbir__simdi_u32
2740{
2741 stbir_uint32 m128i_u32[4];
2742 int m128i_i32[4];
2743 stbir__simdi m128i_i128;
2744} stbir__simdi_u32;
2745
2746static const int STBIR_mask[9] = { 0,0,0,-1,-1,-1,0,0,0 };
2747
2748static const STBIR__SIMDF_CONST(STBIR_max_uint8_as_float, stbir__max_uint8_as_float);
2749static const STBIR__SIMDF_CONST(STBIR_max_uint16_as_float, stbir__max_uint16_as_float);
2750static const STBIR__SIMDF_CONST(STBIR_max_uint8_as_float_inverted, stbir__max_uint8_as_float_inverted);
2751static const STBIR__SIMDF_CONST(STBIR_max_uint16_as_float_inverted, stbir__max_uint16_as_float_inverted);
2752
2753static const STBIR__SIMDF_CONST(STBIR_simd_point5, 0.5f);
2754static const STBIR__SIMDF_CONST(STBIR_ones, 1.0f);
2755static const STBIR__SIMDI_CONST(STBIR_almost_zero, (127 - 13) << 23);
2756static const STBIR__SIMDI_CONST(STBIR_almost_one, 0x3f7fffff);
2757static const STBIR__SIMDI_CONST(STBIR_mantissa_mask, 0xff);
2758static const STBIR__SIMDI_CONST(STBIR_topscale, 0x02000000);
2759
2760// Basically, in simd mode, we unroll the proper amount, and we don't want
2761// the non-simd remnant loops to be unroll because they only run a few times
2762// Adding this switch saves about 5K on clang which is Captain Unroll the 3rd.
2763#define STBIR_SIMD_STREAMOUT_PTR( star ) STBIR_STREAMOUT_PTR( star )
2764#define STBIR_SIMD_NO_UNROLL(ptr) STBIR_NO_UNROLL(ptr)
2765#define STBIR_SIMD_NO_UNROLL_LOOP_START STBIR_NO_UNROLL_LOOP_START
2766#define STBIR_SIMD_NO_UNROLL_LOOP_START_INF_FOR STBIR_NO_UNROLL_LOOP_START_INF_FOR
2767
2768#ifdef STBIR_MEMCPY
2769#undef STBIR_MEMCPY
2770#endif
2771#define STBIR_MEMCPY stbir_simd_memcpy
2772
2773// override normal use of memcpy with much simpler copy (faster and smaller with our sized copies)
2774static void stbir_simd_memcpy( void * dest, void const * src, size_t bytes )
2775{
2776 char STBIR_SIMD_STREAMOUT_PTR (*) d = (char*) dest;
2777 char STBIR_SIMD_STREAMOUT_PTR( * ) d_end = ((char*) dest) + bytes;
2778 char const * s = (char const*) src;
2779
2780 // check overlaps
2781 STBIR_ASSERT( ( ( d >= ( (char*)src) + bytes ) ) || ( ( d + bytes ) <= (char*)src ) );
2782
2783 if ( bytes < (16*stbir__simdfX_float_count) )
2784 {
2785 if ( bytes < 16 )
2786 {
2787 if ( bytes )
2788 {
2789 STBIR_SIMD_NO_UNROLL_LOOP_START
2790 do
2791 {
2792 STBIR_SIMD_NO_UNROLL(d);
2793 d[ 0 ] = s[ 0 ];
2794 ++d;
2795 ++s;
2796 } while ( d < d_end );
2797 }
2798 }
2799 else
2800 {
2801 stbir__simdf x;
2802 // do one unaligned to get us aligned for the stream out below
2803 stbir__simdf_load( x, s );
2804 stbir__simdf_store( d, x );
2805 s += ( ( ( (size_t)d ) + 16 ) & ~15 ) - (size_t)d;
2806 d = (char*)( ( ( (size_t)d ) + 16 ) & ~15 );
2807
2808 STBIR_SIMD_NO_UNROLL_LOOP_START_INF_FOR
2809 for(;;)
2810 {
2811 STBIR_SIMD_NO_UNROLL(d);
2812
2813 if ( d > ( d_end - 16 ) )
2814 {
2815 if ( d == d_end )
2816 return;
2817 s -= d - ( d_end - 16 );
2818 d = d_end - 16;
2819 }
2820
2821 stbir__simdf_load( x, s );
2822 stbir__simdf_store( d, x );
2823 d += 16;
2824 s += 16;
2825 }
2826 }
2827 }
2828 else
2829 {
2830 stbir__simdfX x0,x1,x2,x3;
2831
2832 // do one unaligned to get us aligned for the stream out below
2833 stbir__simdfX_load( x0, s + 0*stbir__simdfX_float_count );
2834 stbir__simdfX_load( x1, s + 4*stbir__simdfX_float_count );
2835 stbir__simdfX_load( x2, s + 8*stbir__simdfX_float_count );
2836 stbir__simdfX_load( x3, s + 12*stbir__simdfX_float_count );
2837 stbir__simdfX_store( d + 0*stbir__simdfX_float_count, x0 );
2838 stbir__simdfX_store( d + 4*stbir__simdfX_float_count, x1 );
2839 stbir__simdfX_store( d + 8*stbir__simdfX_float_count, x2 );
2840 stbir__simdfX_store( d + 12*stbir__simdfX_float_count, x3 );
2841 s += ( ( ( (size_t)d ) + (16*stbir__simdfX_float_count) ) & ~((16*stbir__simdfX_float_count)-1) ) - (size_t)d;
2842 d = (char*)( ( ( (size_t)d ) + (16*stbir__simdfX_float_count) ) & ~((16*stbir__simdfX_float_count)-1) );
2843
2844 STBIR_SIMD_NO_UNROLL_LOOP_START_INF_FOR
2845 for(;;)
2846 {
2847 STBIR_SIMD_NO_UNROLL(d);
2848
2849 if ( d > ( d_end - (16*stbir__simdfX_float_count) ) )
2850 {
2851 if ( d == d_end )
2852 return;
2853 s -= d - ( d_end - (16*stbir__simdfX_float_count) );
2854 d = d_end - (16*stbir__simdfX_float_count);
2855 }
2856
2857 stbir__simdfX_load( x0, s + 0*stbir__simdfX_float_count );
2858 stbir__simdfX_load( x1, s + 4*stbir__simdfX_float_count );
2859 stbir__simdfX_load( x2, s + 8*stbir__simdfX_float_count );
2860 stbir__simdfX_load( x3, s + 12*stbir__simdfX_float_count );
2861 stbir__simdfX_store( d + 0*stbir__simdfX_float_count, x0 );
2862 stbir__simdfX_store( d + 4*stbir__simdfX_float_count, x1 );
2863 stbir__simdfX_store( d + 8*stbir__simdfX_float_count, x2 );
2864 stbir__simdfX_store( d + 12*stbir__simdfX_float_count, x3 );
2865 d += (16*stbir__simdfX_float_count);
2866 s += (16*stbir__simdfX_float_count);
2867 }
2868 }
2869}
2870
2871// float copy that is specifically intentionally overlapping (src is below dest by less than the count, so a forward
2872// copy replicates the first floats through the rest.
2873static void stbir_overlapping_memcpy( float * dest, float const * src, size_t count )
2874{
2875 float STBIR_SIMD_STREAMOUT_PTR (*) sd = (float*) src;
2876 float STBIR_SIMD_STREAMOUT_PTR( * ) s_end = ((float*) src) + count;
2877 ptrdiff_t ofs_to_dest = dest - src;
2878
2879 if ( ofs_to_dest >= 4 ) // is the overlap more than 4 floats away?
2880 {
2881 float STBIR_SIMD_STREAMOUT_PTR( * ) s_end4 = ((float*) src) + (count&~(size_t)3);
2882 STBIR_SIMD_NO_UNROLL_LOOP_START
2883 do
2884 {
2885 stbir__simdf x;
2886 STBIR_SIMD_NO_UNROLL(sd);
2887 stbir__simdf_load( x, sd );
2888 stbir__simdf_store( ( sd + ofs_to_dest ), x );
2889 sd += 4;
2890 } while ( sd < s_end4 );
2891
2892 if ( sd == s_end )
2893 return;
2894 }
2895
2896 do
2897 {
2898 STBIR_SIMD_NO_UNROLL(sd);
2899 sd[ ofs_to_dest ] = sd[ 0 ];
2900 ++sd;
2901 } while ( sd < s_end );
2902}
2903
2904#else // no SSE2
2905
2906// when in scalar mode, we let unrolling happen, so this macro just does the __restrict
2907#define STBIR_SIMD_STREAMOUT_PTR( star ) STBIR_STREAMOUT_PTR( star )
2908#define STBIR_SIMD_NO_UNROLL(ptr)
2909#define STBIR_SIMD_NO_UNROLL_LOOP_START
2910#define STBIR_SIMD_NO_UNROLL_LOOP_START_INF_FOR
2911
2912#endif // SSE2
2913
2914
2915#ifdef STBIR_PROFILE
2916
2917#ifndef STBIR_PROFILE_FUNC
2918
2919#if ( defined(_x86_64) || defined( __x86_64__ ) || defined( _M_X64 ) || defined(__x86_64) || defined(__SSE2__) || defined(STBIR_SSE) || defined( _M_IX86_FP ) || defined(__i386) || defined( __i386__ ) || defined( _M_IX86 ) || defined( _X86_ ) ) && !defined(_M_ARM64EC)
2920
2921#ifdef _MSC_VER
2922
2923 STBIRDEF stbir_uint64 __rdtsc();
2924 #define STBIR_PROFILE_FUNC() __rdtsc()
2925
2926#else // non msvc
2927
2928 static stbir__inline stbir_uint64 STBIR_PROFILE_FUNC()
2929 {
2930 stbir_uint32 lo, hi;
2931 asm volatile ("rdtsc" : "=a" (lo), "=d" (hi) );
2932 return ( ( (stbir_uint64) hi ) << 32 ) | ( (stbir_uint64) lo );
2933 }
2934
2935#endif // msvc
2936
2937#elif defined( STBIR__ARM64 ) || defined(__ARM_NEON__)
2938
2939#if defined( _MSC_VER ) && !defined(__clang__)
2940
2941 #include <intrin.h> // _ReadStatusReg, ARM64_SYSREG
2942 #ifndef ARM64_CNTVCT // winnt.h defines it; without windows.h build the same encoding (CNTVCT_EL0, the generic timer counter)
2943 #define ARM64_CNTVCT ARM64_SYSREG(3,3,14,0,2)
2944 #endif
2945 #define STBIR_PROFILE_FUNC() _ReadStatusReg(ARM64_CNTVCT)
2946
2947#else
2948
2949 static stbir__inline stbir_uint64 STBIR_PROFILE_FUNC()
2950 {
2951 stbir_uint64 tsc;
2952 asm volatile("mrs %0, cntvct_el0" : "=r" (tsc));
2953 return tsc;
2954 }
2955
2956#endif
2957
2958#else // x64, arm
2959
2960#error Unknown platform for profiling.
2961
2962#endif // x64, arm
2963
2964#endif // STBIR_PROFILE_FUNC
2965
2966#define STBIR_ONLY_PROFILE_GET_SPLIT_INFO ,stbir__per_split_info * split_info
2967#define STBIR_ONLY_PROFILE_SET_SPLIT_INFO ,split_info
2968
2969#define STBIR_ONLY_PROFILE_BUILD_GET_INFO ,stbir__info * profile_info
2970#define STBIR_ONLY_PROFILE_BUILD_SET_INFO ,profile_info
2971
2972// super light-weight micro profiler
2973#define STBIR_PROFILE_START_ll( info, wh ) { stbir_uint64 wh##thiszonetime = STBIR_PROFILE_FUNC(); stbir_uint64 * wh##save_parent_excluded_ptr = info->current_zone_excluded_ptr; stbir_uint64 wh##current_zone_excluded = 0; info->current_zone_excluded_ptr = &wh##current_zone_excluded;
2974#define STBIR_PROFILE_END_ll( info, wh ) wh##thiszonetime = STBIR_PROFILE_FUNC() - wh##thiszonetime; info->profile.named.wh += wh##thiszonetime - wh##current_zone_excluded; *wh##save_parent_excluded_ptr += wh##thiszonetime; info->current_zone_excluded_ptr = wh##save_parent_excluded_ptr; }
2975#define STBIR_PROFILE_FIRST_START_ll( info, wh ) { int zone; info->current_zone_excluded_ptr = &info->profile.named.total; for(zone=0;zone<(int)STBIR__ARRAY_SIZE(info->profile.array);zone++) info->profile.array[zone]=0; } STBIR_PROFILE_START_ll( info, wh );
2976#define STBIR_PROFILE_CLEAR_EXTRAS_ll( info, num ) { int extra; for(extra=1;extra<(num);extra++) { int zone; for(zone=0;zone<(int)STBIR__ARRAY_SIZE((info)->profile.array);zone++) (info)[extra].profile.array[zone]=0; } }
2977
2978// for thread data
2979#define STBIR_PROFILE_START( wh ) STBIR_PROFILE_START_ll( split_info, wh )
2980#define STBIR_PROFILE_END( wh ) STBIR_PROFILE_END_ll( split_info, wh )
2981#define STBIR_PROFILE_FIRST_START( wh ) STBIR_PROFILE_FIRST_START_ll( split_info, wh )
2982#define STBIR_PROFILE_CLEAR_EXTRAS() STBIR_PROFILE_CLEAR_EXTRAS_ll( split_info, split_count )
2983#define STBIR_PROFILE_NEST_IN_FIRST( extra ) { if ( ( extra ) != split_info ) ( extra )->current_zone_excluded_ptr = split_info->current_zone_excluded_ptr; } // an extra split's zones count against the first split's open zone
2984
2985// for build data
2986#define STBIR_PROFILE_BUILD_START( wh ) STBIR_PROFILE_START_ll( profile_info, wh )
2987#define STBIR_PROFILE_BUILD_END( wh ) STBIR_PROFILE_END_ll( profile_info, wh )
2988#define STBIR_PROFILE_BUILD_FIRST_START( wh ) STBIR_PROFILE_FIRST_START_ll( profile_info, wh )
2989#define STBIR_PROFILE_BUILD_CLEAR( info ) { int zone; for(zone=0;zone<(int)STBIR__ARRAY_SIZE(info->profile.array);zone++) info->profile.array[zone]=0; }
2990
2991#else // no profile
2992
2993#define STBIR_ONLY_PROFILE_GET_SPLIT_INFO
2994#define STBIR_ONLY_PROFILE_SET_SPLIT_INFO
2995
2996#define STBIR_ONLY_PROFILE_BUILD_GET_INFO
2997#define STBIR_ONLY_PROFILE_BUILD_SET_INFO
2998
2999#define STBIR_PROFILE_START( wh )
3000#define STBIR_PROFILE_END( wh )
3001#define STBIR_PROFILE_FIRST_START( wh )
3002#define STBIR_PROFILE_CLEAR_EXTRAS( )
3003#define STBIR_PROFILE_NEST_IN_FIRST( extra )
3004
3005#define STBIR_PROFILE_BUILD_START( wh )
3006#define STBIR_PROFILE_BUILD_END( wh )
3007#define STBIR_PROFILE_BUILD_FIRST_START( wh )
3008#define STBIR_PROFILE_BUILD_CLEAR( info )
3009
3010#endif // stbir_profile
3011
3012#ifndef STBIR_MEMCPY
3013// For memcpy
3014#include <string.h>
3015#define STBIR_MEMCPY( dest, src, len ) memcpy( dest, src, len )
3016#endif
3017
3018#ifndef STBIR_SIMD
3019
3020// float copy that is specifically intentionally overlapping (src is below dest by less than the count, so a forward
3021// copy replicates the first floats through the rest.
3022static void stbir_overlapping_memcpy( float * dest, float const * src, size_t count )
3023{
3024 float STBIR_SIMD_STREAMOUT_PTR (*) sd = (float*) src;
3025 float STBIR_SIMD_STREAMOUT_PTR( * ) s_end = ((float*) src) + count;
3026 ptrdiff_t ofs_to_dest = dest - src;
3027
3028 if ( ofs_to_dest >= 2 ) // is the overlap more than 2 floats away
3029 {
3030 float STBIR_SIMD_STREAMOUT_PTR( * ) s_end2 = ((float*) src) + (count&~(size_t)1);
3031
3032 STBIR_NO_UNROLL_LOOP_START
3033 do
3034 {
3035 float a,b;
3036 STBIR_NO_UNROLL(sd);
3037 a = sd[0];
3038 b = sd[1];
3039 sd[ ofs_to_dest ] = a;
3040 sd[ ofs_to_dest + 1 ] = b;
3041 sd += 2;
3042 } while ( sd < s_end2 );
3043
3044 if ( sd == s_end )
3045 return;
3046 }
3047
3048 STBIR_NO_UNROLL_LOOP_START
3049 do
3050 {
3051 STBIR_NO_UNROLL(sd);
3052 sd[ ofs_to_dest ] = sd[ 0 ];
3053 ++sd;
3054 } while ( sd < s_end );
3055}
3056
3057#endif
3058
3059// the built-in kernels (all in stbir_real; the generator rounds the weight to float once, on the way into the
3060// coefficient table). The box filter as the overlap of a pixel with the output pixel's footprint: 1 within
3061// 0.5 - scale / 2 of the center, 0 beyond 0.5 + scale / 2, a ramp between.
3062static stbir_real stbir__filter_trapezoid(stbir_real x, stbir_real scale, void * user_data)
3063{
3064 stbir_real halfscale = scale * (stbir_real)0.5;
3065 stbir_real t = (stbir_real)0.5 + halfscale;
3066 stbir_real r = (stbir_real)0.5 - halfscale;
3067 STBIR_ASSERT(scale <= (stbir_real)1.0);
3068 STBIR__UNUSED(user_data);
3069
3070 if ( x < (stbir_real) 0.0 ) x = -x;
3071
3072 if ( x >= t )
3073 return (stbir_real)0.0;
3074 if ( x <= r )
3075 return (stbir_real)1.0;
3076 return ( t - x ) / scale;
3077}
3078
3079// the box's extent depends on the scale, so it is given as a radius (in input pixels): one multiply, so an integer
3080// ratio's tap count is exact
3081static stbir_real stbir__radius_trapezoid(stbir_real scale, stbir_real inv_scale, int is_upsample, void * user_data)
3082{
3083 STBIR__UNUSED(scale);
3084 STBIR__UNUSED(is_upsample);
3085 STBIR__UNUSED(user_data);
3086 return (stbir_real)0.5 * inv_scale + (stbir_real)0.5; // ( 0.5 + scale / 2 ) output pixels downsampling, 0.5 + inv_scale / 2 input pixels upsampling: the same number
3087}
3088
3089static stbir_real stbir__filter_triangle(stbir_real x, stbir_real s, void * user_data)
3090{
3091 STBIR__UNUSED(s);
3092 STBIR__UNUSED(user_data);
3093
3094 if ( x < (stbir_real) 0.0 ) x = -x;
3095
3096 if (x <= (stbir_real) 1.0)
3097 return (stbir_real) 1.0 - x;
3098 else
3099 return (stbir_real) 0.0;
3100}
3101
3102static stbir_real stbir__filter_point(stbir_real x, stbir_real s, void * user_data)
3103{
3104 STBIR__UNUSED(x);
3105 STBIR__UNUSED(s);
3106 STBIR__UNUSED(user_data);
3107
3108 return (stbir_real) 1.0;
3109}
3110
3111// the Mitchell-Netrevalli cubics with C = (1-B)/2: B=1 is the cubic b-spline, 1/3 Mitchell, 0 Catmull-Rom
3112static stbir__inline stbir_real stbir__mitchell_netrevalli(stbir_real x, stbir_real B)
3113{
3114 if ( x < (stbir_real) 0.0 ) x = -x;
3115
3116 if (x < (stbir_real) 1.0)
3117 return (((stbir_real) 6.0 - (stbir_real) 2.0*B) + x*x*(((stbir_real) 9.0 - (stbir_real) 6.0*B)*x - ((stbir_real) 15.0 - (stbir_real) 9.0*B)))/(stbir_real) 6.0;
3118 else if (x < (stbir_real) 2.0)
3119 return (((stbir_real) 12.0 - (stbir_real) 4.0*B) + x*(((stbir_real) 12.0*B - (stbir_real) 24.0) + x*(((stbir_real) 15.0 - (stbir_real) 9.0*B) + ((stbir_real) 2.0*B - (stbir_real) 3.0)*x)))/(stbir_real) 6.0;
3120
3121 return (stbir_real) 0.0;
3122}
3123
3124static stbir_real stbir__filter_cubic(stbir_real x, stbir_real s, void * user_data)
3125{
3126 STBIR__UNUSED(s);
3127 STBIR__UNUSED(user_data);
3128
3129 return stbir__mitchell_netrevalli( x, (stbir_real) 1.0 );
3130}
3131
3132// Catmull-Rom is the family at B=0, written out (it's the upsampling default)
3133static stbir_real stbir__filter_catmullrom(stbir_real x, stbir_real s, void * user_data)
3134{
3135 STBIR__UNUSED(s);
3136 STBIR__UNUSED(user_data);
3137
3138 if ( x < (stbir_real) 0.0 ) x = -x;
3139
3140 if (x < (stbir_real) 1.0)
3141 return (stbir_real) 1.0 - x*x*((stbir_real) 2.5 - (stbir_real) 1.5*x);
3142 else if (x < (stbir_real) 2.0)
3143 return (stbir_real) 2.0 - x*((stbir_real) 4.0 + x*((stbir_real) 0.5*x - (stbir_real) 2.5));
3144
3145 return (stbir_real) 0.0;
3146}
3147
3148static stbir_real stbir__filter_mitchell(stbir_real x, stbir_real s, void * user_data)
3149{
3150 STBIR__UNUSED(s);
3151 STBIR__UNUSED(user_data);
3152
3153 return stbir__mitchell_netrevalli( x, (stbir_real) 1.0 / (stbir_real) 3.0 );
3154}
3155
3156static stbir_real stbir__filter_mitchell_quarter(stbir_real x, stbir_real s, void * user_data)
3157{
3158 STBIR__UNUSED(s);
3159 STBIR__UNUSED(user_data);
3160
3161 return stbir__mitchell_netrevalli( x, (stbir_real) 0.25 );
3162}
3163
3164static stbir_real stbir__filter_mitchell_sixth(stbir_real x, stbir_real s, void * user_data)
3165{
3166 STBIR__UNUSED(s);
3167 STBIR__UNUSED(user_data);
3168
3169 return stbir__mitchell_netrevalli( x, (stbir_real) 1.0 / (stbir_real) 6.0 );
3170}
3171
3172static stbir_real stbir__filter_mitchell_neg_sixth(stbir_real x, stbir_real s, void * user_data)
3173{
3174 STBIR__UNUSED(s);
3175 STBIR__UNUSED(user_data);
3176
3177 return stbir__mitchell_netrevalli( x, (stbir_real) -1.0 / (stbir_real) 6.0 );
3178}
3179
3180static stbir_real stbir__filter_mitchell_neg_third(stbir_real x, stbir_real s, void * user_data)
3181{
3182 STBIR__UNUSED(s);
3183 STBIR__UNUSED(user_data);
3184
3185 return stbir__mitchell_netrevalli( x, (stbir_real) -1.0 / (stbir_real) 3.0 );
3186}
3187
3188// sin(pi x) / (pi x), done here so the coefficients match on every platform (no libm). sin(pi x) flips sign at every
3189// whole x, so reduce to the nearest whole and run the series on [-pi/2, pi/2] (good to ~1e-9).
3190static stbir_real stbir__sinc_pi(stbir_real x)
3191{
3192 stbir_real const pi = (stbir_real) 3.14159265358979323846;
3193 stbir_real u, u2, s;
3194 int n;
3195
3196 if ( x < (stbir_real) 0.0 ) x = -x;
3197
3198 if (x < (stbir_real) 1e-4)
3199 {
3200 stbir_real px = pi * x;
3201 return (stbir_real) 1.0 - px*px / (stbir_real) 6.0;
3202 }
3203
3204 n = (int) ( x + (stbir_real) 0.5 );
3205 u = pi * ( x - (stbir_real) n );
3206 u2 = u*u;
3207 s = (stbir_real) 1.0 - u2 / (stbir_real) 156.0; // the series from the inside out: 1 - u^2 / ( 2k ( 2k + 1 ) )
3208 s = (stbir_real) 1.0 - u2 / (stbir_real) 110.0 * s;
3209 s = (stbir_real) 1.0 - u2 / (stbir_real) 72.0 * s;
3210 s = (stbir_real) 1.0 - u2 / (stbir_real) 42.0 * s;
3211 s = (stbir_real) 1.0 - u2 / (stbir_real) 20.0 * s;
3212 s = (stbir_real) 1.0 - u2 / (stbir_real) 6.0 * s;
3213 s = u * s;
3214 if ( n & 1 ) s = -s;
3215
3216 return s / ( pi * x );
3217}
3218
3219// Lanczos: a sinc windowed by a sinc stretched to the support
3220static stbir_real stbir__filter_lanczos(stbir_real x, stbir_real support)
3221{
3222 if ( x < (stbir_real) 0.0 ) x = -x;
3223
3224 if (x < support)
3225 return stbir__sinc_pi( x ) * stbir__sinc_pi( x / support );
3226
3227 return (stbir_real) 0.0;
3228}
3229
3230static stbir_real stbir__filter_lanczos4(stbir_real x, stbir_real s, void * user_data)
3231{
3232 STBIR__UNUSED(s);
3233 STBIR__UNUSED(user_data);
3234
3235 return stbir__filter_lanczos( x, (stbir_real) 2.0 );
3236}
3237
3238static stbir_real stbir__filter_lanczos5(stbir_real x, stbir_real s, void * user_data)
3239{
3240 STBIR__UNUSED(s);
3241 STBIR__UNUSED(user_data);
3242
3243 return stbir__filter_lanczos( x, (stbir_real) 2.5 );
3244}
3245
3246// Casey Muratori's Bink 2 half pixel kernel ( 1 -4 19 19 -4 1 ) / 32: a Catmull-Rom spline through knots every half
3247// pixel (1 at 0, the weights at the half pixels, 0 at the whole ones), padded at both ends so the four points stay in the table
3248static stbir_real stbir__filter_muratori6(stbir_real x, stbir_real s, void * user_data)
3249{
3250 static stbir_real const knots[9] = { (stbir_real) 19.0/(stbir_real) 32.0, (stbir_real) 1.0, (stbir_real) 19.0/(stbir_real) 32.0, (stbir_real) 0.0, (stbir_real) -4.0/(stbir_real) 32.0, (stbir_real) 0.0, (stbir_real) 1.0/(stbir_real) 32.0, (stbir_real) 0.0, (stbir_real) 0.0 };
3251 stbir_real t, p0, p1, p2, p3;
3252 int i;
3253 STBIR__UNUSED(s);
3254 STBIR__UNUSED(user_data);
3255
3256 if ( x < (stbir_real) 0.0 ) x = -x;
3257
3258 if ( x >= (stbir_real) 3.0 )
3259 return (stbir_real) 0.0;
3260
3261 t = x * (stbir_real) 2.0;
3262 i = (int) t;
3263 t -= (stbir_real) i;
3264
3265 p0 = knots[i];
3266 p1 = knots[i+1];
3267 p2 = knots[i+2];
3268 p3 = knots[i+3];
3269
3270 return (stbir_real) 0.5*((stbir_real) 2.0*p1 + t*((p2 - p0) + t*(((stbir_real) 2.0*p0 - (stbir_real) 5.0*p1 + (stbir_real) 4.0*p2 - p3) + t*(((stbir_real) 3.0*p1 - (stbir_real) 3.0*p2) + (p3 - p0)))));
3271}
3272
3273// point sampling takes the one nearest input pixel in both directions.
3274static stbir_real stbir__radius_point(stbir_real scale, stbir_real inv_scale, int is_upsample, void * user_data)
3275{
3276 STBIR__UNUSED(scale);
3277 STBIR__UNUSED(inv_scale);
3278 STBIR__UNUSED(is_upsample);
3279 STBIR__UNUSED(user_data);
3280 return (stbir_real)0.5;
3281}
3282
3283// the other built-ins' supports: the kernel's radius in filter pixels
3284
3285static stbir_real stbir__support_one(stbir_real scale, void * user_data)
3286{
3287 STBIR__UNUSED(scale);
3288 STBIR__UNUSED(user_data);
3289 return (stbir_real)1.0;
3290}
3291
3292static stbir_real stbir__support_two(stbir_real scale, void * user_data)
3293{
3294 STBIR__UNUSED(scale);
3295 STBIR__UNUSED(user_data);
3296 return (stbir_real)2.0;
3297}
3298
3299static stbir_real stbir__support_two_and_half(stbir_real scale, void * user_data)
3300{
3301 STBIR__UNUSED(scale);
3302 STBIR__UNUSED(user_data);
3303 return (stbir_real)2.5;
3304}
3305
3306static stbir_real stbir__support_three(stbir_real scale, void * user_data)
3307{
3308 STBIR__UNUSED(scale);
3309 STBIR__UNUSED(user_data);
3310 return (stbir_real)3.0;
3311}
3312
3313// the built-in filters, indexed by stbir_filter (everything below STBIR_FILTER_OTHER
3314static stbir__kernel_callback_new * stbir__builtin_kernels[STBIR_FILTER_OTHER] =
3315{
3316 0, // STBIR_FILTER_DEFAULT (resolved before the lookup)
3317 stbir__filter_trapezoid, // STBIR_FILTER_BOX
3318 stbir__filter_triangle, // STBIR_FILTER_TRIANGLE
3319 stbir__filter_cubic, // STBIR_FILTER_CUBICBSPLINE
3320 stbir__filter_catmullrom, // STBIR_FILTER_CATMULLROM
3321 stbir__filter_mitchell, // STBIR_FILTER_MITCHELL
3322 stbir__filter_point, // STBIR_FILTER_POINT_SAMPLE
3323 stbir__filter_mitchell_quarter, // STBIR_FILTER_MITCHELL_QUARTER
3324 stbir__filter_mitchell_sixth, // STBIR_FILTER_MITCHELL_SIXTH
3325 stbir__filter_mitchell_neg_sixth, // STBIR_FILTER_MITCHELL_NEG_SIXTH
3326 stbir__filter_mitchell_neg_third, // STBIR_FILTER_MITCHELL_NEG_THIRD
3327 stbir__filter_lanczos4, // STBIR_FILTER_LANCZOS4
3328 stbir__filter_lanczos5, // STBIR_FILTER_LANCZOS5
3329 stbir__filter_muratori6, // STBIR_FILTER_MURATORI6
3330};
3331
3332static stbir__support_callback_new * stbir__builtin_supports[STBIR_FILTER_OTHER] =
3333{
3334 0, // STBIR_FILTER_DEFAULT
3335 0, // STBIR_FILTER_BOX (a radius)
3336 stbir__support_one, // STBIR_FILTER_TRIANGLE
3337 stbir__support_two, // STBIR_FILTER_CUBICBSPLINE
3338 stbir__support_two, // STBIR_FILTER_CATMULLROM
3339 stbir__support_two, // STBIR_FILTER_MITCHELL
3340 0, // STBIR_FILTER_POINT_SAMPLE (a radius)
3341 stbir__support_two, // STBIR_FILTER_MITCHELL_QUARTER
3342 stbir__support_two, // STBIR_FILTER_MITCHELL_SIXTH
3343 stbir__support_two, // STBIR_FILTER_MITCHELL_NEG_SIXTH
3344 stbir__support_two, // STBIR_FILTER_MITCHELL_NEG_THIRD
3345 stbir__support_two, // STBIR_FILTER_LANCZOS4
3346 stbir__support_two_and_half, // STBIR_FILTER_LANCZOS5
3347 stbir__support_three, // STBIR_FILTER_MURATORI6
3348};
3349
3350static stbir__radius_callback_new * stbir__builtin_radii[STBIR_FILTER_OTHER] =
3351{
3352 0, // STBIR_FILTER_DEFAULT
3353 stbir__radius_trapezoid, // STBIR_FILTER_BOX
3354 0, 0, 0, 0, // STBIR_FILTER_TRIANGLE .. STBIR_FILTER_MITCHELL (supports)
3355 stbir__radius_point, // STBIR_FILTER_POINT_SAMPLE
3356 0, 0, 0, 0, 0, 0, 0, // STBIR_FILTER_MITCHELL_QUARTER .. STBIR_FILTER_MURATORI6 (supports)
3357};
3358
3359// upsample (1:1 included), and a 1:1 with the output on the input's pixel grid (a copy)
3360static int stbir__scale_is_upsample( stbir__scale_info const * scale_info )
3361{
3362 return ( scale_info->scale >= (stbir_real) 1.0 );
3363}
3364
3365static int stbir__scale_is_copy( stbir__scale_info const * scale_info )
3366{
3367 return ( scale_info->positions_exact ) && ( scale_info->scale_numerator == scale_info->scale_denominator );
3368}
3369
3370// the sum under which a coefficient set is zeroed
3371static stbir_real stbir__too_small_coefficient_sum_for( stbir__scale_info const * scale_info )
3372{
3373 if ( stbir__scale_is_upsample( scale_info ) )
3374 return stbir__too_small_coefficient_sum;
3375 return stbir__too_small_coefficient_sum * scale_info->scale;
3376}
3377
3378// floor and ceil without math.h (the header doesn't use it), either sign, for pixel coordinates
3379// and tap counts - the range is known to fit an int
3380static int stbir__floor_to_int( stbir_real x )
3381{
3382 int i = (int) x;
3383 return ( (stbir_real)i > x ) ? i - 1 : i;
3384}
3385
3386static int stbir__ceil_to_int( stbir_real x )
3387{
3388 int i = (int) x;
3389 return ( (stbir_real)i < x ) ? i + 1 : i;
3390}
3391
3392// the filter's radius in input pixels. a radius callback gives it directly. a support callback gives the radius in
3393// filter pixels (called with the ratio <= 1): input pixels when upsampling, output pixels when downsampling, so it
3394// is scaled by inv_scale there. the old float form is converted the same way.
3395static stbir_real stbir__filter_radius_in_input_pixels( stbir__radius_callback_new * radius, stbir__support_callback_new * support_new, stbir__support_callback * support_old, stbir__scale_info const * scale_info, void * user_data )
3396{
3397 int is_upsample = stbir__scale_is_upsample( scale_info );
3398 stbir_real r;
3399
3400 if ( radius )
3401 r = radius( scale_info->scale, scale_info->inv_scale, is_upsample, user_data );
3402 else if ( is_upsample )
3403 {
3404 if ( support_new )
3405 r = support_new( scale_info->inv_scale, user_data );
3406 else
3407 r = (stbir_real) support_old( (float) scale_info->inv_scale, user_data );
3408 }
3409 else
3410 {
3411 if ( support_new )
3412 r = support_new( scale_info->scale, user_data );
3413 else
3414 r = (stbir_real) support_old( (float) scale_info->scale, user_data );
3415 r *= scale_info->inv_scale;
3416 }
3417
3418 // at least half an input pixel (point sampling's), so every output has a tap: the downsample generator walks the
3419 // inputs and gives each output the ones within reach. a user filter's might be under; the built-ins never are
3420 if ( !( r >= (stbir_real)0.5 ) )
3421 r = (stbir_real)0.5;
3422
3423 return r;
3424}
3425
3426static stbir_real stbir__filter_radius_in_output_pixels( stbir__radius_callback_new * radius, stbir__support_callback_new * support_new, stbir__support_callback * support_old, stbir__scale_info const * scale_info, void * user_data )
3427{
3428 return stbir__filter_radius_in_input_pixels( radius, support_new, support_old, scale_info, user_data ) / scale_info->inv_scale;
3429}
3430
3431// This is the maximum number of input samples that can affect an output sample
3432// with the given filter from the output pixel's perspective
3433static int stbir__get_filter_pixel_width( stbir__radius_callback_new * radius, stbir__support_callback_new * support_new, stbir__support_callback * support_old, stbir__scale_info const * scale_info, void * user_data)
3434{
3435 STBIR_ASSERT( ( radius != 0 ) || ( support_new != 0 ) || ( support_old != 0 ) );
3436 return stbir__ceil_to_int( (stbir_real)2.0 * stbir__filter_radius_in_input_pixels( radius, support_new, support_old, scale_info, user_data ) );
3437}
3438
3439// this is how many coefficients per run of the filter (which is different from the filter_pixel_width depending on if we are scattering or gathering)
3440static int stbir__get_coefficient_width(stbir__sampler * samp, int is_gather, void * user_data)
3441{
3442 stbir__scale_info const * scale_info = &samp->scale_info;
3443
3444 switch( is_gather )
3445 {
3446 case 1:
3447 case 2:
3448 // one of slack: a tap exactly on the far edge is in the range (stbir__exact_pixel_range), one more than ceil( 2 x radius )
3449 return stbir__ceil_to_int( (stbir_real)2.0 * stbir__filter_radius_in_input_pixels( samp->filter_radius, samp->filter_support_new, samp->filter_support_old, scale_info, user_data ) ) + 1;
3450 case 0:
3451 return stbir__ceil_to_int( (stbir_real)2.0 * stbir__filter_radius_in_output_pixels( samp->filter_radius, samp->filter_support_new, samp->filter_support_old, scale_info, user_data ) );
3452 default:
3453 STBIR_ASSERT( (is_gather >= 0 ) && (is_gather <= 2 ) );
3454 return 0;
3455 }
3456}
3457
3458static int stbir__get_contributors(stbir__sampler * samp, int is_gather)
3459{
3460 if (is_gather)
3461 return samp->scale_info.output_sub_size;
3462 else
3463 return (samp->scale_info.input_full_size + samp->filter_pixel_margin * 2);
3464}
3465
3466static int stbir__edge_zero_full( int n, int max )
3467{
3468 STBIR__UNUSED(n);
3469 STBIR__UNUSED(max);
3470 return 0; // NOTREACHED
3471}
3472
3473static int stbir__edge_clamp_full( int n, int max )
3474{
3475 if (n < 0)
3476 return 0;
3477
3478 if (n >= max)
3479 return max - 1;
3480
3481 return n; // NOTREACHED
3482}
3483
3484static int stbir__edge_reflect_full( int n, int max )
3485{
3486 if (n < 0)
3487 {
3488 if (n > -max)
3489 return -n;
3490 else
3491 return max - 1;
3492 }
3493
3494 if (n >= max)
3495 {
3496 if ( ( n - max ) >= max ) // (as n >= 2 max, without the overflow)
3497 return 0;
3498 else
3499 return ( max - 1 ) - ( n - max );
3500 }
3501
3502 return n; // NOTREACHED
3503}
3504
3505static int stbir__edge_wrap_full( int n, int max )
3506{
3507 if (n >= 0)
3508 return (n % max);
3509 else
3510 {
3511 int m = (-n) % max;
3512
3513 if (m != 0)
3514 m = max - m;
3515
3516 return (m);
3517 }
3518}
3519
3520typedef int stbir__edge_wrap_func( int n, int max );
3521static stbir__edge_wrap_func * stbir__edge_wrap_slow[] =
3522{
3523 stbir__edge_clamp_full, // STBIR_EDGE_CLAMP
3524 stbir__edge_reflect_full, // STBIR_EDGE_REFLECT
3525 stbir__edge_wrap_full, // STBIR_EDGE_WRAP
3526 stbir__edge_zero_full, // STBIR_EDGE_ZERO
3527};
3528
3529stbir__inline static int stbir__edge_wrap(stbir_edge edge, int n, int max)
3530{
3531 // avoid per-pixel switch
3532 if (n >= 0 && n < max)
3533 return n;
3534 return stbir__edge_wrap_slow[edge]( n, max );
3535}
3536
3537// the ranges' edge handling: in pixel ranges may hang off the image with WRAP (the decoder wraps them, within a width
3538// each side), out pixel ranges are clipped to the output
3539static void stbir__clip_in_pixel_range( int first, int last, int input_size, stbir_edge edge, int * first_pixel, int * last_pixel )
3540{
3541 if ( last < first )
3542 last = first; // point sample mode can span a value *right* at 0.5, and cause these to cross
3543 if ( edge == STBIR_EDGE_WRAP )
3544 {
3545 if ( first < -input_size )
3546 first = -input_size;
3547 if ( ( last - input_size ) >= input_size ) // (as last >= 2 input_size, without the overflow)
3548 last = ( input_size - 1 ) + input_size;
3549 }
3550 *first_pixel = first;
3551 *last_pixel = last;
3552}
3553
3554static void stbir__clip_out_pixel_range( int first, int last, int out_size, int * first_pixel, int * last_pixel )
3555{
3556 if ( first < 0 )
3557 first = 0;
3558 if ( last >= out_size )
3559 last = out_size - 1;
3560 *first_pixel = first;
3561 *last_pixel = last;
3562}
3563
3564// The pixels within support of the center numer / den (quotient kept apart, only the remainder meets the real math).
3565// On the far edge is in, on the near edge out; stbir__tie_epsilon lifts the tie above rounding noise (point sampling
3566// hits both edges at every even ratio). The output range and the real chain ranges below use the same rule.
3567#define stbir__tie_epsilon ( ( sizeof( stbir_real ) == 4 ) ? (stbir_real)( 1.0 / ( 1 << 18 ) ) : (stbir_real)( 1.0 / ( (stbir_int64)1 << 36 ) ) )
3568
3569// The input pixels within out_filter_radius (output pixels) of an output pixel's center.
3570static void stbir__calculate_in_pixel_range( int * first_pixel, int * last_pixel, stbir_real out_pixel_center, stbir_real out_filter_radius, stbir_real inv_scale, stbir_real out_shift, int input_size, stbir_edge edge )
3571{
3572 stbir_real in_pixel_influence_lowerbound = ( out_pixel_center - out_filter_radius + out_shift ) * inv_scale;
3573 stbir_real in_pixel_influence_upperbound = ( out_pixel_center + out_filter_radius + out_shift ) * inv_scale;
3574 stbir__clip_in_pixel_range( stbir__floor_to_int( in_pixel_influence_lowerbound + (stbir_real)0.5 + stbir__tie_epsilon ), stbir__floor_to_int( in_pixel_influence_upperbound - (stbir_real)0.5 + stbir__tie_epsilon ), input_size, edge, first_pixel, last_pixel );
3575}
3576
3577// Output pixel o's center against input pixel i's (in input pixels for upsamples, output pixels for downsamples). With an
3578// exact scale N/D and integer shift it is ( ( 2o + 1 + 2 shift ) D - ( 2i + 1 ) N ) / 2N (or 2D): integers until one
3579// multiply by the precomputed 1 / 2N (or 1 / 2D), exact to float at any coordinate.
3580static stbir_real stbir__tap_offset( stbir__scale_info const * scale_info, int o, int i, int in_input_pixels )
3581{
3582 if ( scale_info->positions_exact )
3583 {
3584 // (the difference is at most the filter's width times 2N or 2D: exact in a double, a float ulp in a float)
3585 stbir_int64 numer = ( 2 * (stbir_int64)o + 1 + 2 * (stbir_int64)scale_info->shift_pixels ) * (stbir_int64)scale_info->scale_denominator
3586 - ( 2 * (stbir_int64)i + 1 ) * (stbir_int64)scale_info->scale_numerator;
3587 return (stbir_real)numer * ( in_input_pixels ? scale_info->tap_scale_in : scale_info->tap_scale_out );
3588 }
3589 if ( in_input_pixels )
3590 return ( ( (stbir_real)o + (stbir_real)0.5 + scale_info->pixel_shift ) * scale_info->inv_scale ) - ( (stbir_real)i + (stbir_real)0.5 );
3591 return ( (stbir_real)o + (stbir_real)0.5 ) - ( ( (stbir_real)i + (stbir_real)0.5 ) * scale_info->scale - scale_info->pixel_shift );
3592}
3593
3594// the kernel's weight at x (in filter pixels): through the new form directly, or an old float kernel's floats
3595static stbir_real stbir__kernel_weight( stbir__kernel_callback_new * kernel_new, stbir__kernel_callback * kernel_old, stbir_real x, stbir_real scale, void * user_data )
3596{
3597 if ( kernel_new )
3598 return kernel_new( x, scale, user_data );
3599 return (stbir_real) kernel_old( (float) x, (float) scale, user_data );
3600}
3601
3602static void stbir__exact_pixel_range( stbir_int64 numer, stbir_int64 den, stbir_real den_reciprocal, stbir_real support, int * first, int * last )
3603{
3604 int q = (int)( numer / den );
3605 stbir_real f = (stbir_real)( numer % den ) * den_reciprocal; // ( -1, 1 ): the same sign as numer
3606 *first = q + stbir__floor_to_int( f - support + (stbir_real)0.5 + stbir__tie_epsilon );
3607 *last = q + stbir__floor_to_int( f + support - (stbir_real)0.5 + stbir__tie_epsilon );
3608}
3609
3610static void stbir__exact_output_range( stbir_int64 numer, stbir_int64 den, stbir_real den_reciprocal, stbir_real support, int * first, int * last )
3611{
3612 int q = (int)( numer / den );
3613 stbir_real f = (stbir_real)( numer % den ) * den_reciprocal;
3614 *first = q + stbir__floor_to_int( f - support - (stbir_real)0.5 - stbir__tie_epsilon ) + 1; // ceil, a tie exactly on it included
3615 *last = q + stbir__floor_to_int( f + support - (stbir_real)0.5 - stbir__tie_epsilon ); // a tie exactly on it excluded
3616}
3617
3618// The input pixels within the filter's support of output pixel o: out_filter_radius is the support in output pixels for
3619// the real chain, support the same in input pixels (the filter's own, for upsamples) for the exact positions - then
3620// the center is the one stbir__tap_offset weights from.
3621static void stbir__output_in_pixel_range( stbir__scale_info const * scale_info, int o, stbir_real out_filter_radius, stbir_real support, stbir_edge edge, int * first_pixel, int * last_pixel )
3622{
3623 if ( scale_info->positions_exact )
3624 {
3625 stbir_int64 numer = ( 2 * (stbir_int64)o + 1 + 2 * (stbir_int64)scale_info->shift_pixels ) * (stbir_int64)scale_info->scale_denominator; // output o's center in input pixels, over 2N
3626 int first, last;
3627 stbir__exact_pixel_range( numer, 2 * (stbir_int64)scale_info->scale_numerator, scale_info->tap_scale_in, support, &first, &last );
3628 stbir__clip_in_pixel_range( first, last, scale_info->input_full_size, edge, first_pixel, last_pixel );
3629 }
3630 else
3631 stbir__calculate_in_pixel_range( first_pixel, last_pixel, (stbir_real)o + (stbir_real)0.5, out_filter_radius, scale_info->inv_scale, scale_info->pixel_shift, scale_info->input_full_size, edge );
3632}
3633
3634static void stbir__calculate_coefficients_for_gather_upsample( stbir_real out_filter_radius, stbir_real support, stbir__kernel_callback_new * kernel_new, stbir__kernel_callback * kernel_old, stbir__scale_info * scale_info, int num_contributors, stbir__contributors* contributors, float* coefficient_group, int coefficient_width, stbir_edge edge, void * user_data )
3635{
3636 int n, end;
3637 int numerator = scale_info->scale_numerator;
3638 int polyphase = ( ( scale_info->scale_is_rational ) && ( numerator < num_contributors ) );
3639
3640 // Looping through out pixels
3641 end = num_contributors; if ( polyphase ) end = numerator;
3642 for (n = 0; n < end; n++)
3643 {
3644 int i;
3645 int last_non_zero;
3646 int in_first_pixel, in_last_pixel;
3647
3648 stbir__output_in_pixel_range( scale_info, n, out_filter_radius, support, edge, &in_first_pixel, &in_last_pixel );
3649
3650 // make sure we never generate a range larger than our precalculated coeff width
3651 // this only happens in point sample mode, but it's a good safe thing to do anyway
3652 if ( ( in_last_pixel - in_first_pixel + 1 ) > coefficient_width )
3653 in_last_pixel = in_first_pixel + coefficient_width - 1;
3654
3655 last_non_zero = -1;
3656 for (i = 0; i <= in_last_pixel - in_first_pixel; i++)
3657 {
3658 stbir_real weight = stbir__kernel_weight( kernel_new, kernel_old, stbir__tap_offset( scale_info, n, i + in_first_pixel, 1 ), scale_info->inv_scale, user_data );
3659 float coeff = (float) weight;
3660
3661 if ( ( weight < stbir__too_small_coefficient ) && ( weight > -stbir__too_small_coefficient ) )
3662 {
3663 if ( i == 0 ) // if we're at the front, just eat zero contributors
3664 {
3665 STBIR_ASSERT ( ( in_last_pixel - in_first_pixel ) != 0 ); // there should be at least one contrib
3666 ++in_first_pixel;
3667 i--;
3668 continue;
3669 }
3670 coeff = 0; // make sure is fully zero (should keep denormals away)
3671 }
3672 else
3673 last_non_zero = i;
3674
3675 coefficient_group[i] = coeff;
3676 }
3677
3678 in_last_pixel = last_non_zero+in_first_pixel; // kills trailing zeros
3679 contributors->n0 = in_first_pixel;
3680 contributors->n1 = in_last_pixel;
3681
3682 STBIR_ASSERT(contributors->n1 >= contributors->n0);
3683
3684 ++contributors;
3685 coefficient_group += coefficient_width;
3686 }
3687}
3688
3689static void stbir__insert_coeff( stbir__contributors * contribs, float * coeffs, int new_pixel, float new_coeff, int max_width )
3690{
3691 if ( contribs->n1 < contribs->n0 ) // this first clause should never happen, but handle in case
3692 {
3693 contribs->n0 = contribs->n1 = new_pixel;
3694 coeffs[0] = new_coeff;
3695 }
3696 else if ( new_pixel <= contribs->n1 ) // before the end
3697 {
3698 if ( new_pixel < contribs->n0 ) // before the front?
3699 {
3700 if ( ( contribs->n1 - new_pixel + 1 ) <= max_width )
3701 {
3702 int j, o = contribs->n0 - new_pixel;
3703 for ( j = contribs->n1 - contribs->n0 ; j >= 0 ; j-- )
3704 coeffs[ j + o ] = coeffs[ j ];
3705 for ( j = 1 ; j < o ; j++ )
3706 coeffs[ j ] = 0;
3707 coeffs[ 0 ] = new_coeff;
3708 contribs->n0 = new_pixel;
3709 }
3710 }
3711 else
3712 {
3713 // add new weight to existing coeff if already there
3714 coeffs[ new_pixel - contribs->n0 ] += new_coeff;
3715 }
3716 }
3717 else
3718 {
3719 if ( ( new_pixel - contribs->n0 + 1 ) <= max_width )
3720 {
3721 int j, e = new_pixel - contribs->n0;
3722 for( j = ( contribs->n1 - contribs->n0 ) + 1 ; j < e ; j++ ) // clear in-betweens coeffs if there are any
3723 coeffs[j] = 0;
3724
3725 coeffs[ e ] = new_coeff;
3726 contribs->n1 = new_pixel;
3727 }
3728 }
3729}
3730
3731// The output pixels within in_pixels_radius (input pixels) of an input pixel's center.
3732static void stbir__calculate_out_pixel_range( int * first_pixel, int * last_pixel, stbir_real in_pixel_center, stbir_real in_pixels_radius, stbir_real scale, stbir_real out_shift, int out_size )
3733{
3734 stbir_real out_pixel_influence_lowerbound = ( in_pixel_center - in_pixels_radius ) * scale - out_shift;
3735 stbir_real out_pixel_influence_upperbound = ( in_pixel_center + in_pixels_radius ) * scale - out_shift;
3736 stbir__clip_out_pixel_range( stbir__floor_to_int( out_pixel_influence_lowerbound - (stbir_real)0.5 - stbir__tie_epsilon ) + 1, stbir__floor_to_int( out_pixel_influence_upperbound - (stbir_real)0.5 - stbir__tie_epsilon ), out_size, first_pixel, last_pixel );
3737}
3738
3739// The output pixels within the filter's support of input pixel i, one pixel at a time - the same ways as
3740// stbir__output_in_pixel_range: in_pixels_radius is the support in input pixels for the real chain, support the
3741// same in output pixels for the exact positions.
3742static void stbir__input_out_pixel_range( stbir__scale_info const * scale_info, int i, stbir_real in_pixels_radius, stbir_real support, int * first_pixel, int * last_pixel )
3743{
3744 if ( scale_info->positions_exact )
3745 {
3746 stbir_int64 numer = ( 2 * (stbir_int64)i + 1 ) * (stbir_int64)scale_info->scale_numerator - 2 * (stbir_int64)scale_info->shift_pixels * (stbir_int64)scale_info->scale_denominator; // input i's center in output pixels, over 2D
3747 int first, last;
3748 stbir__exact_output_range( numer, 2 * (stbir_int64)scale_info->scale_denominator, scale_info->tap_scale_out, support, &first, &last );
3749 stbir__clip_out_pixel_range( first, last, scale_info->output_sub_size, first_pixel, last_pixel );
3750 }
3751 else
3752 stbir__calculate_out_pixel_range( first_pixel, last_pixel, (stbir_real)i + (stbir_real)0.5, in_pixels_radius, scale_info->scale, scale_info->pixel_shift, scale_info->output_sub_size );
3753}
3754
3755static void stbir__calculate_coefficients_for_gather_downsample( int start, int end, stbir_real in_pixels_radius, stbir_real support, stbir__kernel_callback_new * kernel_new, stbir__kernel_callback * kernel_old, stbir__scale_info * scale_info, int coefficient_width, int num_contributors, stbir__contributors * contributors, float * coefficient_group, void * user_data )
3756{
3757 int in_pixel;
3758 int i;
3759 int first_out_inited = -1;
3760 int out_size = scale_info->output_sub_size;
3761 int numerator = scale_info->scale_numerator;
3762 int polyphase = ( ( scale_info->scale_is_rational ) && ( numerator < out_size ) );
3763
3764 STBIR__UNUSED(num_contributors);
3765
3766 // Loop through the input pixels
3767 for (in_pixel = start; in_pixel < end; in_pixel++)
3768 {
3769 int out_first_pixel, out_last_pixel;
3770
3771 stbir__input_out_pixel_range( scale_info, in_pixel, in_pixels_radius, support, &out_first_pixel, &out_last_pixel );
3772
3773 if ( out_first_pixel > out_last_pixel )
3774 continue;
3775
3776 // clamp or exit if we are using polyphase filtering, and the limit is up
3777 if ( polyphase )
3778 {
3779 // when polyphase, you only have to do coeffs up to the numerator count
3780 if ( out_first_pixel >= numerator )
3781 break;
3782
3783 // don't do any extra work, clamp last pixel at numerator too
3784 if ( out_last_pixel >= numerator )
3785 out_last_pixel = numerator - 1;
3786 }
3787
3788 for (i = 0; i <= out_last_pixel - out_first_pixel; i++)
3789 {
3790 stbir_real weight = stbir__kernel_weight( kernel_new, kernel_old, stbir__tap_offset( scale_info, i + out_first_pixel, in_pixel, 0 ), scale_info->scale, user_data );
3791 float coeff = 0.0f;
3792
3793 if ( ( weight >= stbir__too_small_coefficient ) || ( weight <= -stbir__too_small_coefficient ) )
3794 coeff = (float) ( weight * scale_info->scale );
3795
3796 {
3797 int out = i + out_first_pixel;
3798 float * coeffs = coefficient_group + (size_t)out * coefficient_width;
3799 stbir__contributors * contribs = contributors + out;
3800
3801 // is this the first time this output pixel has been seen? Init it.
3802 if ( out > first_out_inited )
3803 {
3804 STBIR_ASSERT( out == ( first_out_inited + 1 ) ); // ensure we have only advanced one at time
3805 first_out_inited = out;
3806 contribs->n0 = in_pixel;
3807 contribs->n1 = in_pixel;
3808 coeffs[0] = coeff;
3809 }
3810 else
3811 {
3812 // insert on end (always in order)
3813 if ( coeffs[0] == 0.0f ) // if the first coefficient is zero, then zap it for this coeff
3814 {
3815 STBIR_ASSERT( ( in_pixel - contribs->n0 ) == 1 ); // ensure that when we zap, we're at the 2nd pos
3816 contribs->n0 = in_pixel;
3817 }
3818 contribs->n1 = in_pixel;
3819 STBIR_ASSERT( ( in_pixel - contribs->n0 ) < coefficient_width );
3820 coeffs[in_pixel - contribs->n0] = coeff;
3821 }
3822 }
3823 }
3824 }
3825}
3826
3827// fold a contributor's out of range taps back into the image for the edge mode, trim its trailing zero weights,
3828// and fold its extents into lowest/highest/widest
3829static stbir__inline void stbir__fold_and_trim_contributor( stbir_edge edge, stbir__contributors * contributor, float * coefficient_group, int coefficient_width, int input_size, stbir__filter_extent_info * extents )
3830{
3831 int input_last_pixel = input_size - 1;
3832 int i;
3833
3834 if ( edge == STBIR_EDGE_ZERO )
3835 {
3836 // In zero edge mode, just remove the out of bounds pixels completely (their weights are accounted for now).
3837 if ( contributor->n1 > input_last_pixel )
3838 contributor->n1 = input_last_pixel;
3839
3840 if ( contributor->n0 < 0 )
3841 {
3842 int skip = -contributor->n0;
3843 int remaining;
3844
3845 contributor->n0 = 0;
3846
3847 // Move the weights down.
3848 remaining = contributor->n1 - contributor->n0 + 1;
3849 for (i = 0; i < remaining; i++)
3850 coefficient_group[i] = coefficient_group[i + skip];
3851 }
3852 }
3853 else if ( ( edge == STBIR_EDGE_CLAMP ) || ( edge == STBIR_EDGE_REFLECT ) )
3854 {
3855 // For clamp and reflect, calculate the true in bounds pixel (based on the edge type) and add the weight to it.
3856
3857 // Right hand side first.
3858 if ( contributor->n1 > input_last_pixel )
3859 {
3860 int first = contributor->n0;
3861 int last = contributor->n1;
3862 contributor->n1 = input_last_pixel;
3863 for (i = input_size; i <= last; i++)
3864 stbir__insert_coeff( contributor, coefficient_group, stbir__edge_wrap_slow[edge]( i, input_size ), coefficient_group[i - first], coefficient_width );
3865 }
3866
3867 // Then the left.
3868 if ( contributor->n0 < 0 )
3869 {
3870 int first = contributor->n0;
3871 float * coeff = coefficient_group - ( first + 1 ); // the weight of pixel -1
3872 float first_coeff;
3873
3874 // Reinsert the weights at their reflected or clamped pixel (insert accumulates if the pixel is already there).
3875 for (i = -1; i > first; i--)
3876 stbir__insert_coeff( contributor, coefficient_group, stbir__edge_wrap_slow[edge]( i, input_size ), *coeff--, coefficient_width );
3877
3878 // Save the first one, since we didn't do it yet (i == n0) - there might be too many coeffs to hold before we shrink.
3879 first_coeff = coeff[0];
3880
3881 // Slide all the coeffs down (we accumulated them into the positive pixels) and reset the first pixel.
3882 contributor->n0 = 0;
3883 for (i = 0; i <= contributor->n1; i++)
3884 coefficient_group[i] = coefficient_group[i - first];
3885
3886 // Now that we have shrunk the contributor, we can insert the first one safely.
3887 stbir__insert_coeff( contributor, coefficient_group, stbir__edge_wrap_slow[edge]( first, input_size ), first_coeff, coefficient_width );
3888 }
3889 }
3890
3891 // Trim trailing zero coefficients, re-zero the unused tail, and widen the extents to this contributor.
3892 if ( contributor->n0 <= contributor->n1 )
3893 {
3894 int count = contributor->n1 - contributor->n0 + 1;
3895 while ( count && ( coefficient_group[count - 1] == 0.0f ) )
3896 --count;
3897
3898 contributor->n1 = contributor->n0 + count - 1;
3899
3900 if ( count > 0 )
3901 {
3902 if ( contributor->n0 < extents->lowest )
3903 extents->lowest = contributor->n0;
3904 if ( contributor->n1 > extents->highest )
3905 extents->highest = contributor->n1;
3906 if ( count > extents->widest )
3907 extents->widest = count;
3908 }
3909
3910 for (i = count; i < coefficient_width; i++)
3911 coefficient_group[i] = 0.0f;
3912 }
3913}
3914
3915static void stbir__cleanup_gathered_coefficients( stbir_edge edge, stbir__filter_extent_info* filter_info, stbir__scale_info * scale_info, int num_contributors, stbir__contributors* contributors, float * coefficients, int coefficient_width )
3916{
3917 int input_size = scale_info->input_full_size;
3918 int numerator = scale_info->scale_numerator;
3919 int denominator = scale_info->scale_denominator;
3920 int polyphase = ( ( scale_info->scale_is_rational ) && ( numerator < num_contributors ) );
3921 int calculated = polyphase ? numerator : num_contributors; // when polyphase, only the first numerator outputs were calculated
3922 stbir_real too_small_sum = stbir__too_small_coefficient_sum_for( scale_info );
3923 int n, i;
3924
3925 // Make the sum of each output's coefficients 1.
3926 for (n = 0; n < calculated; n++)
3927 {
3928 stbir__contributors * contributor = contributors + n;
3929 float * coefficient_group = coefficients + (size_t)n * coefficient_width;
3930 stbir_real total_filter = 0;
3931 int last = contributor->n1 - contributor->n0;
3932
3933 for (i = 0; i <= last; i++)
3934 {
3935 total_filter += (stbir_real) coefficient_group[i];
3936 STBIR_ASSERT( ( coefficient_group[i] >= -2.0f ) && ( coefficient_group[i] <= 2.0f ) ); // check for wonky weights
3937 }
3938
3939 if ( ( total_filter < too_small_sum ) && ( total_filter > -too_small_sum ) )
3940 {
3941 // All coeffs are extremely small, just zero it.
3942 contributor->n1 = contributor->n0;
3943 coefficient_group[0] = 0.0f;
3944 }
3945 else if ( ( total_filter < ( (stbir_real)1.0 - stbir__renorm_tolerance ) ) || ( total_filter > ( (stbir_real)1.0 + stbir__renorm_tolerance ) ) )
3946 {
3947 stbir_real filter_scale = (stbir_real)1.0 / total_filter;
3948 for (i = 0; i <= last; i++)
3949 coefficient_group[i] = (float) ( coefficient_group[i] * filter_scale );
3950 }
3951 }
3952
3953 // Polyphase: the rest of the outputs are the first numerator's, moved along by the denominator once per period.
3954 if ( polyphase )
3955 {
3956 for (n = numerator; n < num_contributors; n++)
3957 {
3958 contributors[n].n0 = contributors[n - numerator].n0 + denominator;
3959 contributors[n].n1 = contributors[n - numerator].n1 + denominator;
3960 }
3961 stbir_overlapping_memcpy( coefficients + (size_t)numerator * coefficient_width, coefficients, (size_t)( num_contributors - numerator ) * coefficient_width );
3962 }
3963
3964 filter_info->lowest = 0x7fffffff;
3965 filter_info->highest = -0x7fffffff;
3966 filter_info->widest = -1;
3967 for (n = 0; n < num_contributors; n++)
3968 stbir__fold_and_trim_contributor( edge, contributors + n, coefficients + (size_t)n * coefficient_width, coefficient_width, input_size, filter_info );
3969}
3970
3971// how many input pixels the horizontal kernel that gets selected for this widest will read
3972static int stbir__horizontal_read_extent( stbir__contributors const * contribs, int widest )
3973{
3974 int mod, extent;
3975
3976 if ( widest <= 17 )
3977 return widest;
3978
3979 mod = widest & 3;
3980 extent = ( ( ( contribs->n1 - contribs->n0 + 1 ) - mod + 3 ) & ~3 ) + mod;
3981 if ( extent < ( 8 + mod ) )
3982 extent = 8 + mod;
3983 return extent;
3984}
3985
3986// the SIMD kernels load a remnant of three coefficients as four, so widths of 4n+3 get a fourth, zero coefficient
3987#define STBIR__COEFFICIENT_PAD( widest ) ( ( (widest) & 3 ) == 3 )
3988#define STBIR__COLUMN_ENTRY_FLOATS( widest ) ( 2 + (widest) + STBIR__COEFFICIENT_PAD( widest ) ) // the entry's size in floats
3989
3990// Repack the coefficient groups from coefficient_width floats each to the run time layout: a stbir__column_entry,
3991// two ints (filled in by stbir__fill_column_entries) followed by widest coefficients.
3992static void stbir__pack_column_entries( int num_entries, float * coefficients, int coefficient_width, int widest )
3993{
3994 int entry_floats = STBIR__COLUMN_ENTRY_FLOATS( widest );
3995 int i, j;
3996
3997 for (i = 0; i < num_entries; i++)
3998 {
3999 // When the entries grow they move up, so do the last one first; otherwise they move down (or stay), first one first.
4000 int n = ( entry_floats > coefficient_width ) ? ( num_entries - 1 - i ) : i;
4001 float * coefficient_group = coefficients + (size_t)n * coefficient_width;
4002 float * entry_coefficients = coefficients + (size_t)n * entry_floats + 2;
4003
4004 if ( entry_coefficients > coefficient_group )
4005 {
4006 for (j = widest - 1; j >= 0; j--)
4007 entry_coefficients[j] = coefficient_group[j];
4008 }
4009 else
4010 {
4011 for (j = 0; j < widest; j++)
4012 entry_coefficients[j] = coefficient_group[j];
4013 }
4014
4015 // The kernels that load one float past the coefficients see a zero.
4016 if ( STBIR__COEFFICIENT_PAD( widest ) )
4017 entry_coefficients[widest] = 0.0f;
4018 }
4019}
4020
4021// the horizontal kernels read a fixed number of pixels from each contributor's n0 (see stbir__horizontal_read_extent),
4022// so contributors near the right edge get moved back (zero weights in front) to keep those reads inside the
4023// decoded range.
4024static void stbir__clip_contributor_reads( int num_contributors, stbir__contributors* contributors, float * coefficients, int stride, int widest, int row0, int row1 )
4025{
4026 int row_end = row1 + 1;
4027 STBIR__UNUSED( row0 ); // only used in an assert
4028
4029 // the fixed width kernels (up to 17 taps) read exactly widest pixels from n0 whatever a contributor's own width is,
4030 // and the generic ones read in fours (stbir__horizontal_read_extent). So we make sure we never read outside the
4031 // decode buffer, by possibly moving the sample area back into the scanline, and putting zero weights first.
4032 // we start on the right edge and check until we're well past the possible
4033 // clip area (2*widest).
4034 {
4035 stbir__contributors * contribs = contributors + num_contributors - 1;
4036 float * coeffs = coefficients + (size_t)stride * ( num_contributors - 1 );
4037
4038 // go until no chance of clipping (this is usually less than 8 lops)
4039 while ( ( contribs >= contributors ) && ( ( contribs->n0 + widest*2 ) >= row_end ) )
4040 {
4041 // might we clip??
4042 if ( ( contribs->n0 + widest ) > row_end )
4043 {
4044 // past 17 taps the generic loops run, and they read to this contributor's own width rounded up to a four,
4045 // not a fixed widest amount - so calculate this
4046 int stop_range = stbir__horizontal_read_extent( contribs, widest );
4047
4048 // now see if we still clip with the refined range
4049 if ( ( contribs->n0 + stop_range ) > row_end )
4050 {
4051 int new_n0 = row_end - stop_range;
4052 int num = contribs->n1 - contribs->n0 + 1;
4053 int backup = contribs->n0 - new_n0;
4054 float * from_co = coeffs + num - 1;
4055 float * to_co = from_co + backup;
4056
4057 STBIR_ASSERT( ( new_n0 >= row0 ) && ( new_n0 < contribs->n0 ) );
4058
4059 // move the coeffs over
4060 while( num )
4061 {
4062 *to_co-- = *from_co--;
4063 --num;
4064 }
4065 // zero new positions
4066 while ( to_co >= coeffs )
4067 *to_co-- = 0;
4068 // set new start point (backing up by less than the mod-4 slack keeps the read extent the same, so this can't clip again)
4069 contribs->n0 = new_n0;
4070 STBIR_ASSERT( ( contribs->n0 + stbir__horizontal_read_extent( contribs, widest ) ) <= row_end );
4071 }
4072 }
4073 --contribs;
4074 coeffs -= stride;
4075 }
4076 }
4077}
4078
4079// pack the coefficient sets into column entries and clip the edge reads
4080static int stbir__pack_coefficients( int num_contributors, stbir__contributors* contributors, float * coefficients, int coefficient_width, int widest, int row0, int row1 )
4081{
4082 stbir__pack_column_entries( num_contributors, coefficients, coefficient_width, widest );
4083 stbir__clip_contributor_reads( num_contributors, contributors, coefficients + 2, STBIR__COLUMN_ENTRY_FLOATS( widest ), widest, row0, row1 );
4084 return widest;
4085}
4086
4087// For polyphase resizing (rational scale resizing), there is a ton of coefficent sharing across the
4088// horizontal scanline. We store the data like this (you need the pro and epi for clamp modes):
4089// [ prologue scanline coeffs ][ big run of repeating coeffs ][ epilogue scanline coeffs ]
4090
4091// Bounds on how many outputs at each edge have taps outside the image (before folding), from the filter geometry.
4092static void stbir__polyphase_edge_bounds( stbir__sampler const * samp, void * user_data, int * out_prologue, int * out_epilogue )
4093{
4094 stbir__scale_info const * scale_info = &samp->scale_info;
4095 int num_contributors = samp->num_contributors;
4096 stbir_real in_pixels_radius;
4097 int prologue = 0;
4098 int epilogue = 0;
4099 int n;
4100
4101 // the reach of an output's taps, plus how far an approximated fraction's copies can drift from these real positions
4102 in_pixels_radius = stbir__filter_radius_in_input_pixels( samp->filter_radius, samp->filter_support_new, samp->filter_support_old, scale_info, user_data ) + (stbir_real)scale_info->approximated_drift;
4103
4104 // Nothing gets folded with wrap.
4105 if ( samp->edge != STBIR_EDGE_WRAP )
4106 {
4107 // Looping through out pixels from the left edge, until one reads nothing before the image
4108 for (n = 0; n < num_contributors; n++)
4109 {
4110 stbir_real in_first_pixel = ( (stbir_real)n + (stbir_real)0.5 + scale_info->pixel_shift ) * scale_info->inv_scale - in_pixels_radius;
4111 if ( stbir__floor_to_int( in_first_pixel + (stbir_real)0.5 ) >= 0 )
4112 break;
4113 ++prologue;
4114 }
4115
4116 // And from the right edge, until one reads nothing past it
4117 for (n = num_contributors - 1; n >= 0; n--)
4118 {
4119 stbir_real in_last_pixel = ( (stbir_real)n + (stbir_real)0.5 + scale_info->pixel_shift ) * scale_info->inv_scale + in_pixels_radius;
4120 if ( stbir__floor_to_int( in_last_pixel - (stbir_real)0.5 ) < scale_info->input_full_size )
4121 break;
4122 ++epilogue;
4123 }
4124
4125 // A pixel of slack each way: this is the real chain, the generators' positions can be exact.
4126 ++prologue;
4127 ++epilogue;
4128 }
4129
4130 *out_prologue = prologue;
4131 *out_epilogue = epilogue;
4132}
4133
4134// for polyphase (rational) scales, we want to unroll the sharing a bit just so we aren't
4135// constantly breaking out of a tight loop to back up the coeff sharing.
4136#define STBIR__POLYPHASE_MIN_BLOCK 256
4137
4138// decide whether this (gather) sampler shares its polyphase coefficient sets, and size its tables accordingly.
4139// called before the allocation pass, so it can only use the sampler's geometry.
4140static void stbir__plan_polyphase_sharing( stbir__sampler * samp, int reads_get_clipped, void * user_data )
4141{
4142 int num_contributors = samp->num_contributors;
4143 int numerator = (int) samp->scale_info.scale_numerator;
4144 int prologue, epilogue, period, repeating;
4145
4146 samp->shared_period = 0;
4147 samp->shared_prologue = 0;
4148 samp->shared_epilogue = 0;
4149
4150 if ( ( !samp->is_gather ) || ( !samp->scale_info.scale_is_rational ) || ( numerator >= num_contributors ) )
4151 return;
4152
4153 stbir__polyphase_edge_bounds( samp, user_data, &prologue, &epilogue );
4154
4155 // the prologue is whole periods, so an output's phase is its index mod numerator
4156 prologue = ( ( prologue + numerator - 1 ) / numerator ) * numerator;
4157
4158 // the horizontal's clip (stbir__clip_contributor_reads) can also move outputs within 2*widest input pixels of the
4159 // right edge: 2*widest*scale outputs, bounded with the coefficient width since widest isn't known yet
4160 if ( reads_get_clipped )
4161 epilogue += (int)( (stbir_real)2.0 * (stbir_real)samp->coefficient_width * samp->scale_info.scale ) + 2;
4162
4163 // the block is one period of sets (one set at period 1, run with an entry stride of zero). the horizontal walks a
4164 // row through it, so its block is padded out to keep the kernel calls long; the vertical fetches a set per row by index
4165 period = numerator;
4166 if ( ( numerator > 1 ) && ( reads_get_clipped ) )
4167 {
4168 while ( period < STBIR__POLYPHASE_MIN_BLOCK )
4169 period += numerator;
4170 }
4171
4172 // not enough repeating outputs to be worth it
4173 repeating = num_contributors - prologue - epilogue;
4174 if ( repeating < period )
4175 return;
4176
4177 samp->shared_period = period;
4178 samp->shared_prologue = prologue;
4179 samp->shared_epilogue = epilogue;
4180 samp->coefficients_size = (size_t)( prologue + period + epilogue ) * STBIR__COLUMN_ENTRY_FLOATS( samp->coefficient_width ) * sizeof(float) + sizeof(float)*STBIR_INPUT_CALLBACK_PADDING;
4181 samp->contributors_size = (size_t)( prologue + period + epilogue + 1 ) * sizeof(stbir__contributors); // one per coefficient set (build time only)
4182}
4183
4184// the shared layout keeps contributors for just the outputs that have their own coefficient sets, in set order:
4185// [ P prologue outputs ][ block outputs ][ E epilogue outputs ].
4186static void stbir__shared_output_range( stbir__sampler const * samp, int x, int * n0, int * n1 )
4187{
4188 int prologue = samp->shared_prologue;
4189 int period = samp->shared_period;
4190 int epilogue = samp->shared_epilogue;
4191 int num_contributors = samp->num_contributors;
4192 stbir__contributors const * contributor;
4193 int shift = 0;
4194
4195 if ( x >= num_contributors - epilogue )
4196 contributor = samp->contributors + prologue + period + ( x - ( num_contributors - epilogue ) );
4197 else if ( x >= prologue )
4198 {
4199 contributor = samp->contributors + prologue + ( ( x - prologue ) % period );
4200 shift = ( ( x - prologue ) / period ) * ( period / (int) samp->scale_info.scale_numerator ) * (int) samp->scale_info.scale_denominator;
4201 }
4202 else
4203 contributor = samp->contributors + x;
4204
4205 *n0 = contributor->n0 + shift;
4206 *n1 = contributor->n1 + shift;
4207}
4208
4209// n0/n1 of output x for either layout
4210static void stbir__output_range( stbir__sampler const * samp, int x, int * n0, int * n1 )
4211{
4212 if ( samp->shared_period )
4213 stbir__shared_output_range( samp, x, n0, n1 );
4214 else
4215 {
4216 *n0 = samp->contributors[ x ].n0;
4217 *n1 = samp->contributors[ x ].n1;
4218 }
4219}
4220
4221// The shared layout's version of stbir__cleanup_gathered_coefficients.
4222static void stbir__cleanup_shared_coefficients( stbir__sampler * samp )
4223{
4224 stbir_edge edge = samp->edge;
4225 stbir__scale_info const * scale_info = &samp->scale_info;
4226 int input_size = scale_info->input_full_size;
4227 int num_contributors = samp->num_contributors;
4228 int coefficient_width = samp->coefficient_width;
4229 int numerator = (int) scale_info->scale_numerator;
4230 int denominator = (int) scale_info->scale_denominator;
4231 int prologue = samp->shared_prologue;
4232 int period = samp->shared_period;
4233 int epilogue = samp->shared_epilogue;
4234 stbir__contributors * contributors = samp->contributors;
4235 float * coefficients = samp->coefficients;
4236 float * phase_coefficients = coefficients + (size_t)prologue * coefficient_width;
4237 stbir__filter_extent_info * extents = &samp->extent_info;
4238 stbir_real too_small_sum = stbir__too_small_coefficient_sum_for( &samp->scale_info );
4239 int n, i;
4240
4241 // Make the sum of each phase's coefficients 1 (as the plain path does for its first numerator outputs).
4242 for (n = 0; n < numerator; n++)
4243 {
4244 stbir__contributors * contributor = contributors + n;
4245 float * coefficient_group = phase_coefficients + (size_t)n * coefficient_width;
4246 stbir_real total_filter = 0;
4247 int last = contributor->n1 - contributor->n0;
4248
4249 for (i = 0; i <= last; i++)
4250 {
4251 total_filter += (stbir_real) coefficient_group[i];
4252 STBIR_ASSERT( ( coefficient_group[i] >= -2.0f ) && ( coefficient_group[i] <= 2.0f ) ); // check for wonky weights
4253 }
4254
4255 if ( ( total_filter < too_small_sum ) && ( total_filter > -too_small_sum ) )
4256 {
4257 // All coeffs are extremely small, just zero it.
4258 contributor->n1 = contributor->n0;
4259 coefficient_group[0] = 0.0f;
4260 }
4261 else if ( ( total_filter < ( (stbir_real)1.0 - stbir__renorm_tolerance ) ) || ( total_filter > ( (stbir_real)1.0 + stbir__renorm_tolerance ) ) )
4262 {
4263 stbir_real filter_scale = (stbir_real)1.0 / total_filter;
4264 for (i = 0; i <= last; i++)
4265 coefficient_group[i] = (float) ( coefficient_group[i] * filter_scale );
4266 }
4267 }
4268
4269 // The contributors for the prologue and period outputs (0..prologue+period-1) are the phase's, moved along by the
4270 // denominator once per period. The epilogue outputs (the last epilogue of the image) follow them.
4271 for (n = numerator; n < prologue + period; n++)
4272 {
4273 contributors[n].n0 = contributors[n - numerator].n0 + denominator;
4274 contributors[n].n1 = contributors[n - numerator].n1 + denominator;
4275 }
4276 for (n = 0; n < epilogue; n++)
4277 {
4278 int out_pixel = num_contributors - epilogue + n;
4279 contributors[prologue + period + n].n0 = contributors[out_pixel % numerator].n0 + ( out_pixel / numerator ) * denominator;
4280 contributors[prologue + period + n].n1 = contributors[out_pixel % numerator].n1 + ( out_pixel / numerator ) * denominator;
4281 }
4282
4283 // The period is the phases repeated; the prologue and epilogue groups start out as copies of their phase.
4284 for (n = numerator; n < period; n++)
4285 STBIR_MEMCPY( phase_coefficients + (size_t)n * coefficient_width, phase_coefficients + (size_t)( n % numerator ) * coefficient_width, coefficient_width * sizeof(float) );
4286 for (n = 0; n < prologue; n++)
4287 STBIR_MEMCPY( coefficients + (size_t)n * coefficient_width, phase_coefficients + (size_t)( n % numerator ) * coefficient_width, coefficient_width * sizeof(float) );
4288 for (n = 0; n < epilogue; n++)
4289 STBIR_MEMCPY( coefficients + (size_t)( prologue + period + n ) * coefficient_width, phase_coefficients + (size_t)( ( num_contributors - epilogue + n ) % numerator ) * coefficient_width, coefficient_width * sizeof(float) );
4290
4291 // Fold and trim every group. The period's outputs are never at an edge (that is what the prologue and epilogue are
4292 // for), so for them this is just the trim, which is the same for every copy of a phase - the period stays periodic.
4293 extents->lowest = 0x7fffffff;
4294 extents->highest = -0x7fffffff;
4295 extents->widest = -1;
4296 for (n = 0; n < prologue + period + epilogue; n++)
4297 {
4298 STBIR_ASSERT( ( n < prologue ) || ( n >= prologue + period ) || ( edge == STBIR_EDGE_WRAP ) || ( ( contributors[n].n0 >= 0 ) && ( contributors[n].n1 < input_size ) ) );
4299 stbir__fold_and_trim_contributor( edge, contributors + n, coefficients + (size_t)n * coefficient_width, coefficient_width, input_size, extents );
4300 }
4301
4302 // lowest and highest are over every output: the extremes are in the prologue and first period, and the last period
4303 // and epilogue.
4304 {
4305 int n0, n1;
4306 stbir__shared_output_range( samp, num_contributors - epilogue - 1, &n0, &n1 );
4307 if ( n1 > extents->highest )
4308 extents->highest = n1;
4309 }
4310}
4311
4312// the shared layout's version of stbir__pack_coefficients
4313static int stbir__pack_shared_coefficients( stbir__sampler * samp, int row0, int row1 )
4314{
4315 int num_contributors = samp->num_contributors;
4316 int widest = samp->extent_info.widest;
4317 int entry_floats = STBIR__COLUMN_ENTRY_FLOATS( widest );
4318 int prologue = samp->shared_prologue;
4319 int period = samp->shared_period;
4320 int epilogue = samp->shared_epilogue;
4321 int n0, n1;
4322
4323 stbir__pack_column_entries( prologue + period + epilogue, samp->coefficients, samp->coefficient_width, widest );
4324 stbir__clip_contributor_reads( epilogue, samp->contributors + prologue + period, samp->coefficients + (size_t)( prologue + period ) * entry_floats + 2, entry_floats, widest, row0, row1 );
4325
4326 // the clipping never needed to reach past the epilogue (the last periodic output is where it would have gone next)
4327 stbir__shared_output_range( samp, num_contributors - epilogue - 1, &n0, &n1 );
4328 STBIR_ASSERT( ( n0 + widest * 2 ) < row1 + 1 );
4329 return widest;
4330}
4331
4332// the packed column entry at an index (the entries are entry_floats apart)
4333static stbir__column_entry * stbir__column_entry_at( stbir__sampler const * samp, int index )
4334{
4335 return (stbir__column_entry *)( samp->coefficients + (size_t)index * samp->entry_floats );
4336}
4337
4338// fill in the column entries' advance and width from the build time contributors (after the entries have been
4339// packed, and after the contributors have been read for the last time - the column slice table). returns 0 when
4340// an advance doesn't fit the entry's int - a horizontal downsample of around 2^29:1 at four channels
4341static int stbir__fill_column_entries( stbir__sampler * samp, int channels )
4342{
4343 stbir__contributors const * contributors = samp->contributors;
4344 int num_entries = samp->shared_period ? ( samp->shared_prologue + samp->shared_period + samp->shared_epilogue ) : samp->num_contributors;
4345 int i;
4346
4347 if ( samp->shared_period )
4348 {
4349 int prologue = samp->shared_prologue;
4350 int period = samp->shared_period;
4351 samp->shared_block_n0 = contributors[prologue].n0;
4352 samp->shared_block_shift = ( period / (int) samp->scale_info.scale_numerator ) * (int) samp->scale_info.scale_denominator;
4353 samp->shared_epilogue_n0 = contributors[prologue + period].n0;
4354 }
4355
4356 // each entry says how far the decode pointer moves to the NEXT column. the block's last entry says how far to the
4357 // block's first column one pass on (the wrap)
4358 for (i = 0; i < num_entries; i++)
4359 {
4360 stbir__column_entry * entry = stbir__column_entry_at( samp, i );
4361 int advance;
4362
4363 if ( ( samp->shared_period ) && ( i == samp->shared_prologue + samp->shared_period - 1 ) )
4364 advance = ( samp->shared_block_n0 + samp->shared_block_shift ) - contributors[i].n0;
4365 else
4366 advance = ( i + 1 < num_entries ) ? ( contributors[i + 1].n0 - contributors[i].n0 ) : 0;
4367
4368 if ( ( (stbir_int64)advance * channels <= -0x7fffffff ) || ( (stbir_int64)advance * channels >= 0x7fffffff ) )
4369 return 0; // error
4370
4371 entry->advance = advance * channels;
4372 entry->width = contributors[i].n1 - contributors[i].n0 + 1;
4373 }
4374 return 1;
4375}
4376
4377// which coefficient set (column entry) output x uses, for either layout
4378static int stbir__output_entry_index( stbir__sampler const * samp, int x )
4379{
4380 if ( samp->shared_period )
4381 {
4382 int prologue = samp->shared_prologue;
4383 int period = samp->shared_period;
4384 int epilogue = samp->shared_epilogue;
4385 int num_contributors = samp->num_contributors;
4386
4387 if ( x >= num_contributors - epilogue )
4388 return prologue + period + ( x - ( num_contributors - epilogue ) );
4389 if ( x >= prologue )
4390 return prologue + ( ( x - prologue ) % period );
4391 }
4392 return x;
4393}
4394
4395static float const * stbir__output_coefficients( stbir__sampler const * samp, int x )
4396{
4397 return samp->coefficients + (size_t)stbir__output_entry_index( samp, x ) * samp->coefficient_width;
4398}
4399
4400// output x's packed column entry
4401static stbir__column_entry const * stbir__output_column_entry( stbir__sampler const * samp, int x )
4402{
4403 return stbir__column_entry_at( samp, stbir__output_entry_index( samp, x ) );
4404}
4405
4406// Vertical scatter functions - used when we are downsampling around 32:1 or more (horizontal is always gather)
4407// advice-wise, if you are downsampling this amount, then you'd better off doing the downsample in a couple steps
4408static void stbir__calculate_coefficients_scatter( stbir__kernel_callback_new * kernel_new, stbir__kernel_callback * kernel_old, stbir__scale_info const * scale_info, int in_pixel, int out_first_pixel, int out_last_pixel, stbir__contributors* contributor, float* coefficient_group, void * user_data )
4409{
4410 int i;
4411
4412 contributor->n0 = out_first_pixel;
4413 contributor->n1 = out_last_pixel;
4414
4415 for (i = 0; i <= out_last_pixel - out_first_pixel; i++)
4416 {
4417 stbir_real weight = stbir__kernel_weight( kernel_new, kernel_old, stbir__tap_offset( scale_info, i + out_first_pixel, in_pixel, 0 ), scale_info->scale, user_data );
4418 float coeff = 0.0f;
4419
4420 if ( ( weight >= stbir__too_small_coefficient ) || ( weight <= -stbir__too_small_coefficient ) )
4421 coeff = (float) ( weight * scale_info->scale );
4422
4423 coefficient_group[i] = coeff;
4424 }
4425}
4426
4427// Normalize each output's coefficients: summed and scaled in stbir_real, each then rounded to float
4428static void stbir__normalize_scatter_coefficients( stbir__contributors* contributors, float* coefficients, int coefficient_width, int num_contributors, int output_size, stbir_real too_small_sum )
4429{
4430 int i, j, first = 0;
4431
4432 for (i = 0; i < output_size; i++)
4433 {
4434 stbir_real total = 0;
4435 stbir_real filter_scale;
4436
4437 while ( ( first < num_contributors ) && ( contributors[first].n1 < i ) )
4438 first++;
4439
4440 for (j = first; j < num_contributors; j++)
4441 {
4442 if (i >= contributors[j].n0 && i <= contributors[j].n1)
4443 total += (stbir_real) coefficients[ (size_t)j * coefficient_width + i - contributors[j].n0 ];
4444 else if (i < contributors[j].n0)
4445 break;
4446 }
4447
4448 if ( ( total < too_small_sum ) && ( total > -too_small_sum ) )
4449 filter_scale = 0; // all coeffs are extremely small, just zero them
4450 else if ( ( total < ( (stbir_real)1.0 - stbir__renorm_tolerance ) ) || ( total > ( (stbir_real)1.0 + stbir__renorm_tolerance ) ) )
4451 filter_scale = (stbir_real)1.0 / total;
4452 else
4453 continue;
4454
4455 for (j = first; j < num_contributors; j++)
4456 {
4457 if (i >= contributors[j].n0 && i <= contributors[j].n1)
4458 coefficients[ (size_t)j * coefficient_width + i - contributors[j].n0 ] = (float) ( coefficients[ (size_t)j * coefficient_width + i - contributors[j].n0 ] * filter_scale );
4459 else if (i < contributors[j].n0)
4460 break;
4461 }
4462 }
4463}
4464
4465// Polyphase: only the first numerator outputs were calculated. From first_copy on, each contributor is the one a
4466// denominator before it with its outputs moved along by the numerator (calculated ones keep their outputs, the copies follow).
4467static void stbir__replicate_scatter_coefficients( stbir__contributors* contributors, float* coefficients, int coefficient_width, int num_contributors, int first_copy, int calculated_end, int numerator, int denominator, int output_size )
4468{
4469 int j, i;
4470
4471 for (j = first_copy; j < num_contributors; j++)
4472 {
4473 stbir__contributors* contributor = contributors + j;
4474 stbir__contributors* source = contributors + j - denominator;
4475 float* coefficient_group = coefficients + (size_t)j * coefficient_width;
4476 float* source_group = coefficients + (size_t)( j - denominator ) * coefficient_width;
4477
4478 if ( j >= calculated_end )
4479 {
4480 contributor->n0 = 0;
4481 contributor->n1 = -1;
4482
4483 // the whole source fits: copy it
4484 if ( source->n1 + numerator < output_size )
4485 {
4486 contributor->n0 = source->n0 + numerator;
4487 contributor->n1 = source->n1 + numerator;
4488 STBIR_MEMCPY( coefficient_group, source_group, (size_t)( source->n1 - source->n0 + 1 ) * sizeof(float) );
4489 continue;
4490 }
4491 }
4492
4493 // append the source's outputs that are in range
4494 for (i = 0; i <= source->n1 - source->n0; i++)
4495 {
4496 int out = source->n0 + i + numerator;
4497 if ( out >= output_size )
4498 break;
4499 if ( contributor->n1 < contributor->n0 )
4500 {
4501 contributor->n0 = contributor->n1 = out;
4502 coefficient_group[0] = source_group[i];
4503 }
4504 else
4505 stbir__insert_coeff( contributor, coefficient_group, out, source_group[i], coefficient_width );
4506 }
4507 }
4508}
4509
4510// The out of range input scanlines: clamp and reflect add their weights to the scanlines they map to (right edge
4511// ascending, then left edge descending - the gather's order), zero drops them, wrap keeps them.
4512static void stbir__fold_scatter_edges( stbir__sampler * samp, int coefficient_width )
4513{
4514 stbir_edge edge = samp->edge;
4515 int input_size = samp->scale_info.input_full_size;
4516 int filter_pixel_margin = samp->filter_pixel_margin;
4517 stbir__contributors* contributors = samp->contributors;
4518 float* coefficients = samp->coefficients;
4519 int n, i;
4520
4521 if ( ( edge == STBIR_EDGE_CLAMP ) || ( edge == STBIR_EDGE_REFLECT ) )
4522 {
4523 int pass;
4524 for (pass = 0; pass < 2; pass++)
4525 {
4526 int from = pass ? -1 : input_size;
4527 int to = pass ? -filter_pixel_margin - 1 : input_size + filter_pixel_margin;
4528 int step = pass ? -1 : 1;
4529 for (n = from; n != to; n += step)
4530 {
4531 stbir__contributors* contributor = contributors + n + filter_pixel_margin;
4532 float* coefficient_group = coefficients + (size_t)( n + filter_pixel_margin ) * coefficient_width;
4533 int folded = stbir__edge_wrap_slow[edge]( n, input_size ) + filter_pixel_margin;
4534
4535 for (i = 0; i <= contributor->n1 - contributor->n0; i++)
4536 stbir__insert_coeff( contributors + folded, coefficients + (size_t)folded * coefficient_width, contributor->n0 + i, coefficient_group[i], coefficient_width );
4537
4538 contributor->n0 = 0;
4539 contributor->n1 = -1;
4540 }
4541 }
4542 }
4543 else if ( edge == STBIR_EDGE_ZERO )
4544 {
4545 for (n = 0; n < filter_pixel_margin; n++)
4546 {
4547 contributors[n].n0 = 0;
4548 contributors[n].n1 = -1;
4549 contributors[input_size + 2 * filter_pixel_margin - 1 - n].n0 = 0;
4550 contributors[input_size + 2 * filter_pixel_margin - 1 - n].n1 = -1;
4551 }
4552 }
4553}
4554
4555// Optimize step - skip zero coefficients at either end, and pack the groups from the build width down to the final width
4556// (an output past the width is dropped). Also notes the first and last input scanlines with any weight.
4557static void stbir__pack_scatter_coefficients( stbir__sampler * samp, int build_width )
4558{
4559 int coefficient_width = samp->coefficient_width;
4560 int filter_pixel_margin = samp->filter_pixel_margin;
4561 int num_contributors = samp->num_contributors;
4562 stbir__contributors* contributors = samp->contributors;
4563 float* coefficients = samp->coefficients;
4564 int j, i;
4565
4566 samp->extent_info.lowest = 0x7fffffff;
4567 samp->extent_info.highest = -0x7fffffff;
4568 samp->extent_info.widest = coefficient_width;
4569
4570 for (j = 0; j < num_contributors; j++)
4571 {
4572 stbir__contributors* contributor = contributors + j;
4573 float const* source_group = coefficients + (size_t)j * build_width;
4574 float* coefficient_group = coefficients + (size_t)j * coefficient_width; // at or before the source, so packing forward in place is safe
4575 int range = contributor->n1 - contributor->n0 + 1;
4576 int skip = 0;
4577
4578 while ( ( skip < range ) && ( source_group[skip] == 0.0f ) )
4579 skip++;
4580 range -= skip;
4581 if ( range > coefficient_width )
4582 range = coefficient_width;
4583 while ( ( range > 0 ) && ( source_group[skip + range - 1] == 0.0f ) )
4584 range--;
4585
4586 for (i = 0; i < range; i++)
4587 coefficient_group[i] = source_group[i + skip];
4588
4589 if ( range <= 0 )
4590 {
4591 contributor->n0 = 0;
4592 contributor->n1 = -1;
4593 continue;
4594 }
4595 contributor->n0 += skip;
4596 contributor->n1 = contributor->n0 + range - 1;
4597
4598 if ( j - filter_pixel_margin < samp->extent_info.lowest )
4599 samp->extent_info.lowest = j - filter_pixel_margin;
4600 samp->extent_info.highest = j - filter_pixel_margin;
4601 }
4602}
4603
4604static void stbir__calculate_scatter_filters( stbir__sampler * samp, stbir_real in_pixels_radius, stbir_real support, void * user_data )
4605{
4606 stbir__scale_info const * scale_info = &samp->scale_info;
4607 int output_size = scale_info->output_sub_size;
4608 int filter_pixel_margin = samp->filter_pixel_margin;
4609 int num_contributors = samp->num_contributors;
4610 int build_width = samp->coefficient_width + 1;
4611 int polyphase = ( scale_info->scale_is_rational ) && ( (int)scale_info->scale_numerator < output_size );
4612 int numerator = polyphase ? (int)scale_info->scale_numerator : output_size;
4613 int calculated_end = num_contributors; // the first contributor with nothing calculated
4614 int first_weighted = -1; // the first contributor with any weight
4615 int n;
4616
4617 // Looping through in pixels
4618 for (n = 0; n < num_contributors; n++)
4619 {
4620 int out_first_pixel, out_last_pixel;
4621
4622 stbir__input_out_pixel_range( scale_info, n - filter_pixel_margin, in_pixels_radius, support, &out_first_pixel, &out_last_pixel );
4623
4624 // when polyphase, you only have to do coeffs up to the numerator count
4625 if ( out_first_pixel >= numerator )
4626 {
4627 calculated_end = n;
4628 break;
4629 }
4630 if ( out_last_pixel >= numerator )
4631 out_last_pixel = numerator - 1;
4632
4633 stbir__calculate_coefficients_scatter( samp->filter_kernel_new, samp->filter_kernel_old, scale_info, n - filter_pixel_margin, out_first_pixel, out_last_pixel, samp->contributors + n, samp->coefficients + (size_t)n * build_width, user_data );
4634
4635 if ( ( first_weighted < 0 ) && ( out_last_pixel >= out_first_pixel ) )
4636 first_weighted = n;
4637 }
4638 for (n = calculated_end; n < num_contributors; n++)
4639 {
4640 samp->contributors[n].n0 = 0;
4641 samp->contributors[n].n1 = -1;
4642 }
4643
4644 stbir__normalize_scatter_coefficients( samp->contributors, samp->coefficients, build_width, calculated_end, numerator, stbir__too_small_coefficient_sum_for( scale_info ) );
4645
4646 // the period is the rational's, which can be a little off the true one, so the copies start where it says
4647 if ( ( polyphase ) && ( first_weighted >= 0 ) )
4648 stbir__replicate_scatter_coefficients( samp->contributors, samp->coefficients, build_width, num_contributors, first_weighted + (int)scale_info->scale_denominator, calculated_end, numerator, (int)scale_info->scale_denominator, output_size );
4649
4650 stbir__fold_scatter_edges( samp, build_width );
4651 stbir__pack_scatter_coefficients( samp, build_width );
4652}
4653
4654static void stbir__calculate_filters( stbir__sampler * samp, void * user_data STBIR_ONLY_PROFILE_BUILD_GET_INFO )
4655{
4656 stbir_real scale = samp->scale_info.scale;
4657 stbir__kernel_callback_new * kernel_new = samp->filter_kernel_new;
4658 stbir__kernel_callback * kernel_old = samp->filter_kernel_old;
4659 stbir_real in_pixels_radius = stbir__filter_radius_in_input_pixels( samp->filter_radius, samp->filter_support_new, samp->filter_support_old, &samp->scale_info, user_data );
4660 stbir_real inv_scale = samp->scale_info.inv_scale;
4661 int input_full_size = samp->scale_info.input_full_size;
4662 int gather_num_contributors = samp->num_contributors;
4663 stbir__contributors* gather_contributors = samp->contributors;
4664 float * gather_coeffs = samp->coefficients;
4665 int gather_coefficient_width = samp->coefficient_width;
4666
4667 // with shared polyphase sets the generated phases go after the prologue sets
4668 if ( samp->shared_period )
4669 gather_coeffs += (size_t)samp->shared_prologue * gather_coefficient_width;
4670
4671 switch ( samp->is_gather )
4672 {
4673 case 1: // gather upsample
4674 {
4675 stbir_real filter_support = in_pixels_radius; // the filter's own pixels are input pixels
4676 stbir_real out_pixels_radius = filter_support * scale;
4677
4678 stbir__calculate_coefficients_for_gather_upsample( out_pixels_radius, filter_support, kernel_new, kernel_old, &samp->scale_info, gather_num_contributors, gather_contributors, gather_coeffs, gather_coefficient_width, samp->edge, user_data );
4679
4680 STBIR_PROFILE_BUILD_START( cleanup );
4681 if ( samp->shared_period )
4682 stbir__cleanup_shared_coefficients( samp );
4683 else
4684 stbir__cleanup_gathered_coefficients( samp->edge, &samp->extent_info, &samp->scale_info, gather_num_contributors, gather_contributors, gather_coeffs, gather_coefficient_width );
4685 STBIR_PROFILE_BUILD_END( cleanup );
4686 }
4687 break;
4688
4689 case 0: // scatter downsample (only on vertical)
4690 case 2: // gather downsample
4691 {
4692 stbir_real filter_support = in_pixels_radius / inv_scale; // in output pixels
4693 int filter_pixel_margin = samp->filter_pixel_margin;
4694 int input_end = input_full_size + filter_pixel_margin;
4695
4696 if ( !samp->is_gather )
4697 {
4698 STBIR_PROFILE_BUILD_START( scatter );
4699 stbir__calculate_scatter_filters( samp, in_pixels_radius, filter_support, user_data );
4700 STBIR_PROFILE_BUILD_END( scatter );
4701 break;
4702 }
4703
4704 stbir__calculate_coefficients_for_gather_downsample( -filter_pixel_margin, input_end, in_pixels_radius, filter_support, kernel_new, kernel_old, &samp->scale_info, gather_coefficient_width, gather_num_contributors, gather_contributors, gather_coeffs, user_data );
4705
4706 STBIR_PROFILE_BUILD_START( cleanup );
4707 if ( samp->shared_period )
4708 stbir__cleanup_shared_coefficients( samp );
4709 else
4710 stbir__cleanup_gathered_coefficients( samp->edge, &samp->extent_info, &samp->scale_info, gather_num_contributors, gather_contributors, gather_coeffs, gather_coefficient_width );
4711 STBIR_PROFILE_BUILD_END( cleanup );
4712 }
4713 break;
4714 }
4715}
4716
4717
4718
4719//========================================================================================================
4720// scanline decoders and encoders
4721
4722#define stbir__coder_min_num 1
4723#define STB_IMAGE_RESIZE_DO_CODERS
4724#include STBIR__HEADER_FILENAME
4725
4726#define stbir__decode_suffix BGRA
4727#define stbir__decode_swizzle
4728#define stbir__decode_order0 2
4729#define stbir__decode_order1 1
4730#define stbir__decode_order2 0
4731#define stbir__decode_order3 3
4732#define stbir__encode_order0 2
4733#define stbir__encode_order1 1
4734#define stbir__encode_order2 0
4735#define stbir__encode_order3 3
4736#define stbir__coder_min_num 4
4737#define STB_IMAGE_RESIZE_DO_CODERS
4738#include STBIR__HEADER_FILENAME
4739
4740#define stbir__decode_suffix ARGB
4741#define stbir__decode_swizzle
4742#define stbir__decode_order0 1
4743#define stbir__decode_order1 2
4744#define stbir__decode_order2 3
4745#define stbir__decode_order3 0
4746#define stbir__encode_order0 3
4747#define stbir__encode_order1 0
4748#define stbir__encode_order2 1
4749#define stbir__encode_order3 2
4750#define stbir__coder_min_num 4
4751#define STB_IMAGE_RESIZE_DO_CODERS
4752#include STBIR__HEADER_FILENAME
4753
4754#define stbir__decode_suffix ABGR
4755#define stbir__decode_swizzle
4756#define stbir__decode_order0 3
4757#define stbir__decode_order1 2
4758#define stbir__decode_order2 1
4759#define stbir__decode_order3 0
4760#define stbir__encode_order0 3
4761#define stbir__encode_order1 2
4762#define stbir__encode_order2 1
4763#define stbir__encode_order3 0
4764#define stbir__coder_min_num 4
4765#define STB_IMAGE_RESIZE_DO_CODERS
4766#include STBIR__HEADER_FILENAME
4767
4768#define stbir__decode_suffix AR
4769#define stbir__decode_swizzle
4770#define stbir__decode_order0 1
4771#define stbir__decode_order1 0
4772#define stbir__decode_order2 3
4773#define stbir__decode_order3 2
4774#define stbir__encode_order0 1
4775#define stbir__encode_order1 0
4776#define stbir__encode_order2 3
4777#define stbir__encode_order3 2
4778#define stbir__coder_min_num 2
4779#define STB_IMAGE_RESIZE_DO_CODERS
4780#include STBIR__HEADER_FILENAME
4781
4782
4783// fancy alpha means we expand to keep both premultipied and non-premultiplied color channels
4784static void stbir__fancy_alpha_weight_4ch( float * out_buffer, ptrdiff_t width_times_channels )
4785{
4786 float STBIR_STREAMOUT_PTR(*) out = out_buffer;
4787 float const * end_decode = out_buffer + ( width_times_channels / 4 ) * 7; // decode buffer aligned to end of out_buffer
4788 float STBIR_STREAMOUT_PTR(*) decode = (float*)end_decode - width_times_channels;
4789
4790 // fancy alpha is stored internally as R G B A Rpm Gpm Bpm
4791
4792 #ifdef STBIR_SIMD
4793
4794 #ifdef STBIR_SIMD8
4795 decode += 16;
4796 STBIR_NO_UNROLL_LOOP_START
4797 while ( decode <= end_decode )
4798 {
4799 stbir__simdf8 d0,d1,a0,a1,p0,p1;
4800 STBIR_NO_UNROLL(decode);
4801 stbir__simdf8_load( d0, decode-16 );
4802 stbir__simdf8_load( d1, decode-16+8 );
4803 stbir__simdf8_0123to33333333( a0, d0 );
4804 stbir__simdf8_0123to33333333( a1, d1 );
4805 stbir__simdf8_mult( p0, a0, d0 );
4806 stbir__simdf8_mult( p1, a1, d1 );
4807 stbir__simdf8_bot4s( a0, d0, p0 );
4808 stbir__simdf8_bot4s( a1, d1, p1 );
4809 stbir__simdf8_top4s( d0, d0, p0 );
4810 stbir__simdf8_top4s( d1, d1, p1 );
4811 stbir__simdf8_store ( out, a0 );
4812 stbir__simdf8_store ( out+7, d0 );
4813 stbir__simdf8_store ( out+14, a1 );
4814 stbir__simdf8_store ( out+21, d1 );
4815 decode += 16;
4816 out += 28;
4817 }
4818 decode -= 16;
4819 #else
4820 decode += 8;
4821 STBIR_NO_UNROLL_LOOP_START
4822 while ( decode <= end_decode )
4823 {
4824 stbir__simdf d0,a0,d1,a1,p0,p1;
4825 STBIR_NO_UNROLL(decode);
4826 stbir__simdf_load( d0, decode-8 );
4827 stbir__simdf_load( d1, decode-8+4 );
4828 stbir__simdf_0123to3333( a0, d0 );
4829 stbir__simdf_0123to3333( a1, d1 );
4830 stbir__simdf_mult( p0, a0, d0 );
4831 stbir__simdf_mult( p1, a1, d1 );
4832 stbir__simdf_store ( out, d0 );
4833 stbir__simdf_store ( out+4, p0 );
4834 stbir__simdf_store ( out+7, d1 );
4835 stbir__simdf_store ( out+7+4, p1 );
4836 decode += 8;
4837 out += 14;
4838 }
4839 decode -= 8;
4840 #endif
4841
4842 // might be one last odd pixel
4843 #ifdef STBIR_SIMD8
4844 STBIR_NO_UNROLL_LOOP_START
4845 while ( decode < end_decode )
4846 #else
4847 if ( decode < end_decode )
4848 #endif
4849 {
4850 stbir__simdf d,a,p;
4851 STBIR_NO_UNROLL(decode);
4852 stbir__simdf_load( d, decode );
4853 stbir__simdf_0123to3333( a, d );
4854 stbir__simdf_mult( p, a, d );
4855 stbir__simdf_store ( out, d );
4856 stbir__simdf_store ( out+4, p );
4857 decode += 4;
4858 out += 7;
4859 }
4860
4861 #else
4862
4863 while( decode < end_decode )
4864 {
4865 float r = decode[0], g = decode[1], b = decode[2], alpha = decode[3];
4866 out[0] = r;
4867 out[1] = g;
4868 out[2] = b;
4869 out[3] = alpha;
4870 out[4] = r * alpha;
4871 out[5] = g * alpha;
4872 out[6] = b * alpha;
4873 out += 7;
4874 decode += 4;
4875 }
4876
4877 #endif
4878}
4879
4880static void stbir__fancy_alpha_weight_2ch( float * out_buffer, ptrdiff_t width_times_channels )
4881{
4882 float STBIR_STREAMOUT_PTR(*) out = out_buffer;
4883 float const * end_decode = out_buffer + ( width_times_channels / 2 ) * 3;
4884 float STBIR_STREAMOUT_PTR(*) decode = (float*)end_decode - width_times_channels;
4885
4886 // for fancy alpha, turns into: [X A Xpm][X A Xpm],etc
4887
4888 #ifdef STBIR_SIMD
4889
4890 decode += 8;
4891 if ( decode <= end_decode )
4892 {
4893 STBIR_NO_UNROLL_LOOP_START
4894 do {
4895 #ifdef STBIR_SIMD8
4896 stbir__simdf8 d0,a0,p0;
4897 STBIR_NO_UNROLL(decode);
4898 stbir__simdf8_load( d0, decode-8 );
4899 stbir__simdf8_0123to11331133( p0, d0 );
4900 stbir__simdf8_0123to00220022( a0, d0 );
4901 stbir__simdf8_mult( p0, p0, a0 );
4902
4903 stbir__simdf_store2( out, stbir__if_simdf8_cast_to_simdf4( d0 ) );
4904 stbir__simdf_store( out+2, stbir__if_simdf8_cast_to_simdf4( p0 ) );
4905 stbir__simdf_store2h( out+3, stbir__if_simdf8_cast_to_simdf4( d0 ) );
4906
4907 stbir__simdf_store2( out+6, stbir__simdf8_gettop4( d0 ) );
4908 stbir__simdf_store( out+8, stbir__simdf8_gettop4( p0 ) );
4909 stbir__simdf_store2h( out+9, stbir__simdf8_gettop4( d0 ) );
4910 #else
4911 stbir__simdf d0,a0,d1,a1,p0,p1;
4912 STBIR_NO_UNROLL(decode);
4913 stbir__simdf_load( d0, decode-8 );
4914 stbir__simdf_load( d1, decode-8+4 );
4915 stbir__simdf_0123to1133( p0, d0 );
4916 stbir__simdf_0123to1133( p1, d1 );
4917 stbir__simdf_0123to0022( a0, d0 );
4918 stbir__simdf_0123to0022( a1, d1 );
4919 stbir__simdf_mult( p0, p0, a0 );
4920 stbir__simdf_mult( p1, p1, a1 );
4921
4922 stbir__simdf_store2( out, d0 );
4923 stbir__simdf_store( out+2, p0 );
4924 stbir__simdf_store2h( out+3, d0 );
4925
4926 stbir__simdf_store2( out+6, d1 );
4927 stbir__simdf_store( out+8, p1 );
4928 stbir__simdf_store2h( out+9, d1 );
4929 #endif
4930 decode += 8;
4931 out += 12;
4932 } while ( decode <= end_decode );
4933 }
4934 decode -= 8;
4935 #endif
4936
4937 STBIR_SIMD_NO_UNROLL_LOOP_START
4938 while( decode < end_decode )
4939 {
4940 float x = decode[0], y = decode[1];
4941 STBIR_SIMD_NO_UNROLL(decode);
4942 out[0] = x;
4943 out[1] = y;
4944 out[2] = x * y;
4945 out += 3;
4946 decode += 2;
4947 }
4948}
4949
4950static void stbir__fancy_alpha_unweight_4ch( float * encode_buffer, ptrdiff_t width_times_channels )
4951{
4952 float STBIR_SIMD_STREAMOUT_PTR(*) encode = encode_buffer;
4953 float STBIR_SIMD_STREAMOUT_PTR(*) input = encode_buffer;
4954 float const * end_output = encode_buffer + width_times_channels;
4955
4956 // fancy RGBA is stored internally as R G B A Rpm Gpm Bpm
4957
4958 STBIR_SIMD_NO_UNROLL_LOOP_START
4959 do {
4960 float alpha = input[3];
4961#ifdef STBIR_SIMD
4962 stbir__simdf i,ia;
4963 STBIR_SIMD_NO_UNROLL(encode);
4964 if ( alpha < stbir__alpha_too_small_to_divide )
4965 {
4966 stbir__simdf_load( i, input );
4967 stbir__simdf_store( encode, i );
4968 }
4969 else
4970 {
4971 stbir__simdf_load1frep4( ia, 1.0f / alpha );
4972 stbir__simdf_load( i, input+4 );
4973 stbir__simdf_mult( i, i, ia );
4974 stbir__simdf_store( encode, i );
4975 encode[3] = alpha;
4976 }
4977#else
4978 if ( alpha < stbir__alpha_too_small_to_divide )
4979 {
4980 encode[0] = input[0];
4981 encode[1] = input[1];
4982 encode[2] = input[2];
4983 }
4984 else
4985 {
4986 float ialpha = 1.0f / alpha;
4987 encode[0] = input[4] * ialpha;
4988 encode[1] = input[5] * ialpha;
4989 encode[2] = input[6] * ialpha;
4990 }
4991 encode[3] = alpha;
4992#endif
4993
4994 input += 7;
4995 encode += 4;
4996 } while ( encode < end_output );
4997}
4998
4999// format: [X A Xpm][X A Xpm] etc
5000static void stbir__fancy_alpha_unweight_2ch( float * encode_buffer, ptrdiff_t width_times_channels )
5001{
5002 float STBIR_SIMD_STREAMOUT_PTR(*) encode = encode_buffer;
5003 float STBIR_SIMD_STREAMOUT_PTR(*) input = encode_buffer;
5004 float const * end_output = encode_buffer + width_times_channels;
5005
5006 do {
5007 float alpha = input[1];
5008 encode[0] = input[0];
5009 if ( alpha >= stbir__alpha_too_small_to_divide )
5010 encode[0] = input[2] / alpha;
5011 encode[1] = alpha;
5012
5013 input += 3;
5014 encode += 2;
5015 } while ( encode < end_output );
5016}
5017
5018static void stbir__simple_alpha_weight_4ch( float * decode_buffer, ptrdiff_t width_times_channels )
5019{
5020 float STBIR_STREAMOUT_PTR(*) decode = decode_buffer;
5021 float const * end_decode = decode_buffer + width_times_channels;
5022
5023 #ifdef STBIR_SIMD
5024 {
5025 decode += 2 * stbir__simdfX_float_count;
5026 STBIR_NO_UNROLL_LOOP_START
5027 while ( decode <= end_decode )
5028 {
5029 stbir__simdfX d0,a0,d1,a1;
5030 STBIR_NO_UNROLL(decode);
5031 stbir__simdfX_load( d0, decode-2*stbir__simdfX_float_count );
5032 stbir__simdfX_load( d1, decode-2*stbir__simdfX_float_count+stbir__simdfX_float_count );
5033 stbir__simdfX_aaa1( a0, d0, STBIR_onesX );
5034 stbir__simdfX_aaa1( a1, d1, STBIR_onesX );
5035 stbir__simdfX_mult( d0, d0, a0 );
5036 stbir__simdfX_mult( d1, d1, a1 );
5037 stbir__simdfX_store ( decode-2*stbir__simdfX_float_count, d0 );
5038 stbir__simdfX_store ( decode-2*stbir__simdfX_float_count+stbir__simdfX_float_count, d1 );
5039 decode += 2 * stbir__simdfX_float_count;
5040 }
5041 decode -= 2 * stbir__simdfX_float_count;
5042
5043 // few last pixels remnants
5044 #ifdef STBIR_SIMD8
5045 STBIR_NO_UNROLL_LOOP_START
5046 while ( decode < end_decode )
5047 #else
5048 if ( decode < end_decode )
5049 #endif
5050 {
5051 stbir__simdf d,a;
5052 stbir__simdf_load( d, decode );
5053 stbir__simdf_aaa1( a, d, STBIR__CONSTF(STBIR_ones) );
5054 stbir__simdf_mult( d, d, a );
5055 stbir__simdf_store ( decode, d );
5056 decode += 4;
5057 }
5058 }
5059
5060 #else
5061
5062 while( decode < end_decode )
5063 {
5064 float alpha = decode[3];
5065 decode[0] *= alpha;
5066 decode[1] *= alpha;
5067 decode[2] *= alpha;
5068 decode += 4;
5069 }
5070
5071 #endif
5072}
5073
5074static void stbir__simple_alpha_weight_2ch( float * decode_buffer, ptrdiff_t width_times_channels )
5075{
5076 float STBIR_STREAMOUT_PTR(*) decode = decode_buffer;
5077 float const * end_decode = decode_buffer + width_times_channels;
5078
5079 #ifdef STBIR_SIMD
5080 decode += 2 * stbir__simdfX_float_count;
5081 STBIR_NO_UNROLL_LOOP_START
5082 while ( decode <= end_decode )
5083 {
5084 stbir__simdfX d0,a0,d1,a1;
5085 STBIR_NO_UNROLL(decode);
5086 stbir__simdfX_load( d0, decode-2*stbir__simdfX_float_count );
5087 stbir__simdfX_load( d1, decode-2*stbir__simdfX_float_count+stbir__simdfX_float_count );
5088 stbir__simdfX_a1a1( a0, d0, STBIR_onesX );
5089 stbir__simdfX_a1a1( a1, d1, STBIR_onesX );
5090 stbir__simdfX_mult( d0, d0, a0 );
5091 stbir__simdfX_mult( d1, d1, a1 );
5092 stbir__simdfX_store ( decode-2*stbir__simdfX_float_count, d0 );
5093 stbir__simdfX_store ( decode-2*stbir__simdfX_float_count+stbir__simdfX_float_count, d1 );
5094 decode += 2 * stbir__simdfX_float_count;
5095 }
5096 decode -= 2 * stbir__simdfX_float_count;
5097 #endif
5098
5099 STBIR_SIMD_NO_UNROLL_LOOP_START
5100 while( decode < end_decode )
5101 {
5102 float alpha = decode[1];
5103 STBIR_SIMD_NO_UNROLL(decode);
5104 decode[0] *= alpha;
5105 decode += 2;
5106 }
5107}
5108
5109static void stbir__simple_alpha_unweight_4ch( float * encode_buffer, ptrdiff_t width_times_channels )
5110{
5111 float STBIR_SIMD_STREAMOUT_PTR(*) encode = encode_buffer;
5112 float const * end_output = encode_buffer + width_times_channels;
5113
5114 STBIR_SIMD_NO_UNROLL_LOOP_START
5115 do {
5116 float alpha = encode[3];
5117
5118#ifdef STBIR_SIMD
5119 stbir__simdf i,ia;
5120 STBIR_SIMD_NO_UNROLL(encode);
5121 if ( alpha >= stbir__alpha_too_small_to_divide )
5122 {
5123 stbir__simdf_load1frep4( ia, 1.0f / alpha );
5124 stbir__simdf_load( i, encode );
5125 stbir__simdf_mult( i, i, ia );
5126 stbir__simdf_store( encode, i );
5127 encode[3] = alpha;
5128 }
5129#else
5130 if ( alpha >= stbir__alpha_too_small_to_divide )
5131 {
5132 float ialpha = 1.0f / alpha;
5133 encode[0] *= ialpha;
5134 encode[1] *= ialpha;
5135 encode[2] *= ialpha;
5136 }
5137#endif
5138 encode += 4;
5139 } while ( encode < end_output );
5140}
5141
5142static void stbir__simple_alpha_unweight_2ch( float * encode_buffer, ptrdiff_t width_times_channels )
5143{
5144 float STBIR_SIMD_STREAMOUT_PTR(*) encode = encode_buffer;
5145 float const * end_output = encode_buffer + width_times_channels;
5146
5147 do {
5148 float alpha = encode[1];
5149 if ( alpha >= stbir__alpha_too_small_to_divide )
5150 encode[0] /= alpha;
5151 encode += 2;
5152 } while ( encode < end_output );
5153}
5154
5155
5156// only used in RGB->BGR or BGR->RGB
5157static void stbir__simple_flip_3ch( float * decode_buffer, ptrdiff_t width_times_channels )
5158{
5159 float STBIR_STREAMOUT_PTR(*) decode = decode_buffer;
5160 float const * end_decode = decode_buffer + width_times_channels;
5161
5162#ifdef STBIR_SIMD
5163 #ifdef stbir__simdf_swiz2 // do we have two argument swizzles?
5164 end_decode -= 12;
5165 STBIR_NO_UNROLL_LOOP_START
5166 while( decode <= end_decode )
5167 {
5168 // on arm64 8 instructions, no overlapping stores
5169 stbir__simdf a,b,c,na,nb;
5170 STBIR_SIMD_NO_UNROLL(decode);
5171 stbir__simdf_load( a, decode );
5172 stbir__simdf_load( b, decode+4 );
5173 stbir__simdf_load( c, decode+8 );
5174
5175 na = stbir__simdf_swiz2( a, b, 2, 1, 0, 5 );
5176 b = stbir__simdf_swiz2( a, b, 4, 3, 6, 7 );
5177 nb = stbir__simdf_swiz2( b, c, 0, 1, 4, 3 );
5178 c = stbir__simdf_swiz2( b, c, 2, 7, 6, 5 );
5179
5180 stbir__simdf_store( decode, na );
5181 stbir__simdf_store( decode+4, nb );
5182 stbir__simdf_store( decode+8, c );
5183 decode += 12;
5184 }
5185 end_decode += 12;
5186 #else
5187 end_decode -= 24;
5188 STBIR_NO_UNROLL_LOOP_START
5189 while( decode <= end_decode )
5190 {
5191 // 26 instructions on x64
5192 stbir__simdf a,b,c,d,e,f,g;
5193 float i21, i23;
5194 STBIR_SIMD_NO_UNROLL(decode);
5195 stbir__simdf_load( a, decode );
5196 stbir__simdf_load( b, decode+3 );
5197 stbir__simdf_load( c, decode+6 );
5198 stbir__simdf_load( d, decode+9 );
5199 stbir__simdf_load( e, decode+12 );
5200 stbir__simdf_load( f, decode+15 );
5201 stbir__simdf_load( g, decode+18 );
5202
5203 a = stbir__simdf_swiz( a, 2, 1, 0, 3 );
5204 b = stbir__simdf_swiz( b, 2, 1, 0, 3 );
5205 c = stbir__simdf_swiz( c, 2, 1, 0, 3 );
5206 d = stbir__simdf_swiz( d, 2, 1, 0, 3 );
5207 e = stbir__simdf_swiz( e, 2, 1, 0, 3 );
5208 f = stbir__simdf_swiz( f, 2, 1, 0, 3 );
5209 g = stbir__simdf_swiz( g, 2, 1, 0, 3 );
5210
5211 // stores overlap, need to be in order,
5212 stbir__simdf_store( decode, a );
5213 i21 = decode[21];
5214 stbir__simdf_store( decode+3, b );
5215 i23 = decode[23];
5216 stbir__simdf_store( decode+6, c );
5217 stbir__simdf_store( decode+9, d );
5218 stbir__simdf_store( decode+12, e );
5219 stbir__simdf_store( decode+15, f );
5220 stbir__simdf_store( decode+18, g );
5221 decode[21] = i23;
5222 decode[23] = i21;
5223 decode += 24;
5224 }
5225 end_decode += 24;
5226 #endif
5227#else
5228 end_decode -= 12;
5229 STBIR_NO_UNROLL_LOOP_START
5230 while( decode <= end_decode )
5231 {
5232 // 16 instructions
5233 float t0,t1,t2,t3;
5234 STBIR_NO_UNROLL(decode);
5235 t0 = decode[0];
5236 t1 = decode[3];
5237 t2 = decode[6];
5238 t3 = decode[9];
5239 decode[0] = decode[2];
5240 decode[3] = decode[5];
5241 decode[6] = decode[8];
5242 decode[9] = decode[11];
5243 decode[2] = t0;
5244 decode[5] = t1;
5245 decode[8] = t2;
5246 decode[11] = t3;
5247 decode += 12;
5248 }
5249 end_decode += 12;
5250#endif
5251
5252 STBIR_NO_UNROLL_LOOP_START
5253 while( decode < end_decode )
5254 {
5255 float t = decode[0];
5256 STBIR_NO_UNROLL(decode);
5257 decode[0] = decode[2];
5258 decode[2] = t;
5259 decode += 3;
5260 }
5261}
5262
5263
5264// decodes the input pixels that a column slice needs for input scanline n into output_buffer
5265// most of the complexity in this function is for wrap mode, we need to grab from multiple
5266// points in the input buffer
5267static void stbir__decode_scanline(stbir__info const * stbir_info, int n, float * output_buffer, stbir__column_slice const * slice STBIR_ONLY_PROFILE_GET_SPLIT_INFO )
5268{
5269 int channels = stbir_info->channels;
5270 int effective_channels = stbir_info->effective_channels;
5271 int input_sample_in_bytes = stbir__type_size[stbir_info->input_type] * channels;
5272 stbir_edge edge_horizontal = stbir_info->horizontal.edge;
5273 stbir_edge edge_vertical = stbir_info->vertical.edge;
5274 int input_full_size = stbir_info->horizontal.scale_info.input_full_size;
5275 int row = stbir__edge_wrap(edge_vertical, n, stbir_info->vertical.scale_info.input_full_size);
5276 const void* input_plane_data = ( (char *) stbir_info->input_data ) + (ptrdiff_t)( row * stbir_info->input_stride_bytes );
5277 float * full_decode_buffer = output_buffer - (ptrdiff_t)slice->in_n0 * effective_channels; // decode buffer is indexed in input pixel space, so back it up to input pixel zero (the products are 64-bit: a tile of a huge input can start past 2^31 / channels)
5278 float * last_decoded = output_buffer;
5279 int run_n0[3], run_n1[3], run_input_x[3];
5280 int num_runs = 0, r;
5281 int n0 = slice->in_n0, n1 = slice->in_n1;
5282
5283 // if we are on edge_zero, and we get in here with an out of bounds n, then the calculate filters has failed
5284 STBIR_ASSERT( !(edge_vertical == STBIR_EDGE_ZERO && (n < 0 || n >= stbir_info->vertical.scale_info.input_full_size)) );
5285
5286 if ( edge_horizontal == STBIR_EDGE_WRAP )
5287 {
5288 // the margins are never wider than the scanline (stbir__set_sampler clamps them), so each side wraps exactly once
5289 STBIR_ASSERT( n0 >= -input_full_size );
5290 if ( n1 > ( ( input_full_size - 1 ) + input_full_size ) )
5291 n1 = ( input_full_size - 1 ) + input_full_size; // the conservative range can poke one past what the filter can reach, that pixel just gets zeroed below ( 2 x 2^30 - 1, written so it can't overflow )
5292
5293 if ( n0 < 0 )
5294 {
5295 run_n0[num_runs] = n0;
5296 run_n1[num_runs] = ( n1 < -1 ) ? n1 : -1;
5297 run_input_x[num_runs] = n0 + input_full_size;
5298 ++num_runs;
5299 }
5300 if ( ( n1 >= 0 ) && ( n0 < input_full_size ) )
5301 {
5302 run_n0[num_runs] = ( n0 > 0 ) ? n0 : 0;
5303 run_n1[num_runs] = ( n1 < ( input_full_size - 1 ) ) ? n1 : ( input_full_size - 1 );
5304 run_input_x[num_runs] = run_n0[num_runs];
5305 ++num_runs;
5306 }
5307 if ( n1 >= input_full_size )
5308 {
5309 run_n0[num_runs] = ( n0 > input_full_size ) ? n0 : input_full_size;
5310 run_n1[num_runs] = n1;
5311 run_input_x[num_runs] = run_n0[num_runs] - input_full_size;
5312 ++num_runs;
5313 }
5314 }
5315 else
5316 {
5317 STBIR_ASSERT( ( n0 >= 0 ) && ( n1 < input_full_size ) );
5318 run_n0[0] = n0;
5319 run_n1[0] = n1;
5320 run_input_x[0] = n0;
5321 num_runs = 1;
5322 }
5323
5324 for( r = 0 ; r < num_runs ; r++ )
5325 {
5326 int width = run_n1[r] + 1 - run_n0[r];
5327 ptrdiff_t width_times_channels = (ptrdiff_t)width * channels;
5328 float * decode_buffer = full_decode_buffer + (ptrdiff_t)run_n0[r] * effective_channels;
5329 float * end_decode = full_decode_buffer + (ptrdiff_t)( run_n1[r] + 1 ) * effective_channels;
5330 void const * input_data;
5331
5332 // read directly out of input plane by default
5333 input_data = ( (char*)input_plane_data ) + (ptrdiff_t)run_input_x[r] * input_sample_in_bytes;
5334
5335 // if we have an input callback, call it to get the input data
5336 if ( stbir_info->in_pixels_cb )
5337 {
5338 // call the callback with a temp buffer (that they can choose to use or not). the temp is just right aligned memory in the decode_buffer itself
5339 input_data = stbir_info->in_pixels_cb( ( (char*) end_decode ) - ( (ptrdiff_t)width * input_sample_in_bytes ) + stbir_info->input_callback_offset_bytes, input_plane_data, width, run_input_x[r], row, stbir_info->user_data );
5340 }
5341
5342 STBIR_PROFILE_START( decode );
5343 // convert the pixels info the float decode_buffer, (we index from end_decode, so that when channels<effective_channels, we are right justified in the buffer)
5344 last_decoded = stbir_info->decode_pixels( (float*)end_decode - width_times_channels, width_times_channels, input_data );
5345 STBIR_PROFILE_END( decode );
5346
5347 if (stbir_info->alpha_weight)
5348 {
5349 STBIR_PROFILE_START( alpha );
5350 stbir_info->alpha_weight( decode_buffer, width_times_channels );
5351 STBIR_PROFILE_END( alpha );
5352 }
5353 }
5354
5355 // the horizontal kernels sometimes read a few samples past the input end, so
5356 // zero the extra to avoid NaN (has to happen after decoding, since the input
5357 // buffer might overwrite them. Should just be a couple clears.
5358 {
5359 float * range_end = full_decode_buffer + (ptrdiff_t)( slice->in_n1 + 1 ) * effective_channels + 2;
5360 while ( last_decoded < range_end )
5361 *last_decoded++ = 0.0f;
5362 }
5363}
5364
5365// advance to the next set of coeffs and contrib info
5366#define stbir__next_contributor() \
5367 decode_buffer += column->advance; \
5368 column = ( (stbir__column_entry const *)( ( (char const *)(column) ) + (entry_stride) ) );
5369
5370
5371//=================
5372// Do 1 channel horizontal routines
5373
5374#ifdef STBIR_SIMD
5375
5376#define stbir__1_coeff_only() \
5377 stbir__simdf tot,c; \
5378 STBIR_SIMD_NO_UNROLL(decode); \
5379 stbir__simdf_load1( c, hc ); \
5380 stbir__simdf_mult1_mem( tot, c, decode );
5381
5382#define stbir__2_coeff_only() \
5383 stbir__simdf tot,c,d; \
5384 STBIR_SIMD_NO_UNROLL(decode); \
5385 stbir__simdf_load2z( c, hc ); \
5386 stbir__simdf_load2( d, decode ); \
5387 stbir__simdf_mult( tot, c, d ); \
5388 stbir__simdf_0123to1230( c, tot ); \
5389 stbir__simdf_add1( tot, tot, c );
5390
5391#define stbir__3_coeff_only() \
5392 stbir__simdf tot,c,t; \
5393 STBIR_SIMD_NO_UNROLL(decode); \
5394 stbir__simdf_load( c, hc ); \
5395 stbir__simdf_mult_mem( tot, c, decode ); \
5396 stbir__simdf_0123to1230( c, tot ); \
5397 stbir__simdf_0123to2301( t, tot ); \
5398 stbir__simdf_add1( tot, tot, c ); \
5399 stbir__simdf_add1( tot, tot, t );
5400
5401#define stbir__store_output_tiny() \
5402 stbir__simdf_store1( output, tot ); \
5403 stbir__next_contributor(); \
5404 output += 1;
5405
5406#define stbir__4_coeff_start() \
5407 stbir__simdf tot,c; \
5408 STBIR_SIMD_NO_UNROLL(decode); \
5409 stbir__simdf_load( c, hc ); \
5410 stbir__simdf_mult_mem( tot, c, decode ); \
5411
5412#define stbir__4_coeff_continue_from_4( ofs ) \
5413 STBIR_SIMD_NO_UNROLL(decode); \
5414 stbir__simdf_load( c, hc + (ofs) ); \
5415 stbir__simdf_madd_mem( tot, tot, c, decode+(ofs) );
5416
5417#define stbir__1_coeff_remnant( ofs ) \
5418 { stbir__simdf d; \
5419 stbir__simdf_load1z( c, hc + (ofs) ); \
5420 stbir__simdf_load1( d, decode + (ofs) ); \
5421 stbir__simdf_madd( tot, tot, d, c ); }
5422
5423#define stbir__2_coeff_remnant( ofs ) \
5424 { stbir__simdf d; \
5425 stbir__simdf_load2z( c, hc+(ofs) ); \
5426 stbir__simdf_load2( d, decode+(ofs) ); \
5427 stbir__simdf_madd( tot, tot, d, c ); }
5428
5429#define stbir__3_coeff_setup() \
5430 stbir__simdf mask; \
5431 stbir__simdf_load( mask, STBIR_mask + 3 );
5432
5433#define stbir__3_coeff_remnant( ofs ) \
5434 stbir__simdf_load( c, hc+(ofs) ); \
5435 stbir__simdf_and( c, c, mask ); \
5436 stbir__simdf_madd_mem( tot, tot, c, decode+(ofs) );
5437
5438// four outputs per iteration for one channel - do 4 1-channel versions
5439#define stbir__x4_setup() \
5440 stbir__column_entry const * e1 = (stbir__column_entry const *)( (char const *)column + entry_stride ); \
5441 stbir__column_entry const * e2 = (stbir__column_entry const *)( (char const *)e1 + entry_stride ); \
5442 stbir__column_entry const * e3 = (stbir__column_entry const *)( (char const *)e2 + entry_stride ); \
5443 float const * d0 = decode_buffer, * d1 = d0 + column->advance, * d2 = d1 + e1->advance, * d3 = d2 + e2->advance; \
5444 float const * h0 = column->coefficients, * h1 = e1->coefficients, * h2 = e2->coefficients, * h3 = e3->coefficients; \
5445 stbir__simdf a0, a1, a2, a3, c;
5446
5447#define stbir__x4_start() \
5448 do { \
5449 stbir__simdf_load( c, h0 ); \
5450 stbir__simdf_mult_mem( a0, c, d0 ); \
5451 stbir__simdf_load( c, h1 ); \
5452 stbir__simdf_mult_mem( a1, c, d1 ); \
5453 stbir__simdf_load( c, h2 ); \
5454 stbir__simdf_mult_mem( a2, c, d2 ); \
5455 stbir__simdf_load( c, h3 ); \
5456 stbir__simdf_mult_mem( a3, c, d3 ); \
5457 } while(0)
5458
5459#define stbir__x4_cont( ofs ) \
5460 do { \
5461 stbir__simdf_load( c, h0 + (ofs) ); \
5462 stbir__simdf_madd_mem( a0, a0, c, d0 + (ofs) ); \
5463 stbir__simdf_load( c, h1 + (ofs) ); \
5464 stbir__simdf_madd_mem( a1, a1, c, d1 + (ofs) ); \
5465 stbir__simdf_load( c, h2 + (ofs) ); \
5466 stbir__simdf_madd_mem( a2, a2, c, d2 + (ofs) ); \
5467 stbir__simdf_load( c, h3 + (ofs) ); \
5468 stbir__simdf_madd_mem( a3, a3, c, d3 + (ofs) ); \
5469 } while(0)
5470
5471#define stbir__x4_rem1( ofs ) \
5472 do { \
5473 stbir__simdf d; \
5474 stbir__simdf_load1z( c, h0 + (ofs) ); \
5475 stbir__simdf_load1( d, d0 + (ofs) ); \
5476 stbir__simdf_madd( a0, a0, d, c ); \
5477 stbir__simdf_load1z( c, h1 + (ofs) ); \
5478 stbir__simdf_load1( d, d1 + (ofs) ); \
5479 stbir__simdf_madd( a1, a1, d, c ); \
5480 stbir__simdf_load1z( c, h2 + (ofs) ); \
5481 stbir__simdf_load1( d, d2 + (ofs) ); \
5482 stbir__simdf_madd( a2, a2, d, c ); \
5483 stbir__simdf_load1z( c, h3 + (ofs) ); \
5484 stbir__simdf_load1( d, d3 + (ofs) ); \
5485 stbir__simdf_madd( a3, a3, d, c ); \
5486 } while(0)
5487
5488#define stbir__x4_rem2( ofs ) \
5489 do { \
5490 stbir__simdf d; \
5491 stbir__simdf_load2z( c, h0 + (ofs) ); \
5492 stbir__simdf_load2( d, d0 + (ofs) ); \
5493 stbir__simdf_madd( a0, a0, d, c ); \
5494 stbir__simdf_load2z( c, h1 + (ofs) ); \
5495 stbir__simdf_load2( d, d1 + (ofs) ); \
5496 stbir__simdf_madd( a1, a1, d, c ); \
5497 stbir__simdf_load2z( c, h2 + (ofs) ); \
5498 stbir__simdf_load2( d, d2 + (ofs) ); \
5499 stbir__simdf_madd( a2, a2, d, c ); \
5500 stbir__simdf_load2z( c, h3 + (ofs) ); \
5501 stbir__simdf_load2( d, d3 + (ofs) ); \
5502 stbir__simdf_madd( a3, a3, d, c ); \
5503 } while(0)
5504
5505// widths of 4n+3 carry a zero after their last coefficient (STBIR__COEFFICIENT_PAD), so three is a full four
5506#define stbir__x4_rem3( ofs ) stbir__x4_cont( ofs )
5507
5508// two coefficients: ( d0 d0+1 | d1 d1+1 ) * ( c0 c0+1 | c1 c1+1 ), twice, then one pairwise add gives the four outputs
5509#define stbir__x4_2taps() \
5510 do { \
5511 stbir__simdf_load2( c, h0 ); \
5512 stbir__simdf_load2hmerge( c, c, h1 ); \
5513 stbir__simdf_load2( a0, d0 ); \
5514 stbir__simdf_load2hmerge( a0, a0, d1 ); \
5515 stbir__simdf_mult( a0, a0, c ); \
5516 stbir__simdf_load2( a2, h2 ); \
5517 stbir__simdf_load2hmerge( a2, a2, h3 ); \
5518 stbir__simdf_load2( a3, d2 ); \
5519 stbir__simdf_load2hmerge( a3, a3, d3 ); \
5520 stbir__simdf_mult( a1, a3, a2 ); \
5521 stbir__simdf_hadd2( a0, a0, a1 ); \
5522 } while(0)
5523
5524// the same association as the per pixel path's reduction, ( l0 + l2 ) + ( l1 + l3 ), so the results match it bit for bit
5525#define stbir__x4_finish() \
5526 do { \
5527 stbir__simdf_addhalves2( a0, a0, a1 ); \
5528 stbir__simdf_addhalves2( a2, a2, a3 ); \
5529 stbir__simdf_hadd2( a0, a0, a2 ); \
5530 } while(0)
5531
5532// the three coefficient per pixel path reduces as ( l0 + l1 ) + l2, which the pairwise tree reproduces (lane 3 is the zero pad)
5533#define stbir__x4_finish3() \
5534 do { \
5535 stbir__simdf_hadd2( a0, a0, a1 ); \
5536 stbir__simdf_hadd2( a2, a2, a3 ); \
5537 stbir__simdf_hadd2( a0, a0, a2 ); \
5538 } while(0)
5539
5540#define stbir__x4_store() \
5541 do { \
5542 stbir__simdf_store( output, a0 ); \
5543 decode_buffer = d3 + e3->advance; \
5544 column = (stbir__column_entry const *)( (char const *)e3 + entry_stride ); \
5545 output += 4; \
5546 } while(0)
5547
5548// polyphase period 1 for 1 channels (when have exact downsamples 2:1, 3:1, etc - common for mipmaps)
5549#define stbir__period1_setup_4() stbir__simdf P0; stbir__simdf_load( P0, hc );
5550#define stbir__period1_setup_5() stbir__simdf P0, PR; stbir__simdf_load( P0, hc ); stbir__simdf_load1z( PR, hc + 4 );
5551#define stbir__period1_setup_8() stbir__simdf P0, P1; stbir__simdf_load( P0, hc ); stbir__simdf_load( P1, hc + 4 );
5552#define stbir__period1_setup_9() stbir__simdf P0, P1, PR; stbir__simdf_load( P0, hc ); stbir__simdf_load( P1, hc + 4 ); stbir__simdf_load1z( PR, hc + 8 );
5553#define stbir__period1_setup_11() stbir__simdf P0, P1, PR3; stbir__simdf_load( P0, hc ); stbir__simdf_load( P1, hc + 4 ); stbir__simdf_load( PR3, hc + 8 ); // the fourth of PR3 is the 4n+3 zero pad
5554#define stbir__period1_setup_12() stbir__simdf P0, P1, P2; stbir__simdf_load( P0, hc ); stbir__simdf_load( P1, hc + 4 ); stbir__simdf_load( P2, hc + 8 );
5555#define stbir__period1_setup_13() stbir__simdf PR; stbir__period1_setup_12() stbir__simdf_load1z( PR, hc + 12 );
5556#define stbir__period1_setup_16() stbir__simdf P3; stbir__period1_setup_12() stbir__simdf_load( P3, hc + 12 );
5557#define stbir__period1_setup_17() stbir__simdf PR; stbir__period1_setup_16() stbir__simdf_load1z( PR, hc + 16 );
5558
5559#define stbir__period1_4_coeff_start() \
5560 stbir__simdf tot, c; \
5561 STBIR_SIMD_NO_UNROLL(decode); \
5562 stbir__simdf_mult_mem( tot, P0, decode );
5563
5564#define stbir__period1_4_coeff_continue() \
5565 stbir__simdf_madd_mem( tot, tot, P1, decode + 4 );
5566#define stbir__period1_4_coeff_continue2() \
5567 stbir__simdf_madd_mem( tot, tot, P2, decode + 8 );
5568#define stbir__period1_4_coeff_continue3() \
5569 stbir__simdf_madd_mem( tot, tot, P3, decode + 12 );
5570
5571#define stbir__period1_1_coeff_remnant( ofs ) \
5572 do { \
5573 stbir__simdf d; \
5574 stbir__simdf_load1( d, decode + (ofs) ); \
5575 stbir__simdf_madd( tot, tot, d, PR ); \
5576 } while(0)
5577#define stbir__period1_3_coeff_remnant( ofs ) \
5578 stbir__simdf_madd_mem( tot, tot, PR3, decode + (ofs) );
5579
5580#define stbir__period1_store_output() \
5581 stbir__simdf_0123to2301( c, tot ); \
5582 stbir__simdf_add( tot, tot, c ); \
5583 stbir__simdf_0123to1230( c, tot ); \
5584 stbir__simdf_add1( tot, tot, c ); \
5585 stbir__simdf_store1( output, tot ); \
5586 decode_buffer += advance; \
5587 output += 1;
5588
5589// the four outputs' decode pointers are one base plus multiples of the (constant) advance
5590#define stbir__period1_x4_setup() \
5591 float const * d0 = decode_buffer; \
5592 int const dofs1 = advance, dofs2 = advance * 2, dofs3 = advance * 3; \
5593 stbir__simdf a0, a1, a2, a3;
5594
5595#define stbir__period1_x4_start() \
5596 do { \
5597 stbir__simdf_mult_mem( a0, P0, d0 ); \
5598 stbir__simdf_mult_mem( a1, P0, d0 + dofs1 ); \
5599 stbir__simdf_mult_mem( a2, P0, d0 + dofs2 ); \
5600 stbir__simdf_mult_mem( a3, P0, d0 + dofs3 ); \
5601 } while(0)
5602
5603// the next four coefficients, for each of the four outputs
5604#define stbir__period1_x4_next( P, ofs ) \
5605 do { \
5606 stbir__simdf_madd_mem( a0, a0, P, d0 + (ofs) ); \
5607 stbir__simdf_madd_mem( a1, a1, P, d0 + dofs1 + (ofs) ); \
5608 stbir__simdf_madd_mem( a2, a2, P, d0 + dofs2 + (ofs) ); \
5609 stbir__simdf_madd_mem( a3, a3, P, d0 + dofs3 + (ofs) ); \
5610 } while(0)
5611#define stbir__period1_x4_continue() stbir__period1_x4_next( P1, 4 )
5612#define stbir__period1_x4_continue2() stbir__period1_x4_next( P2, 8 )
5613#define stbir__period1_x4_continue3() stbir__period1_x4_next( P3, 12 )
5614#define stbir__period1_x4_remnant3( ofs ) stbir__period1_x4_next( PR3, ofs )
5615
5616#define stbir__period1_x4_remnant( ofs ) \
5617 do { \
5618 stbir__simdf d; \
5619 stbir__simdf_load1( d, d0 + (ofs) ); \
5620 stbir__simdf_madd( a0, a0, d, PR ); \
5621 stbir__simdf_load1( d, d0 + dofs1 + (ofs) ); \
5622 stbir__simdf_madd( a1, a1, d, PR ); \
5623 stbir__simdf_load1( d, d0 + dofs2 + (ofs) ); \
5624 stbir__simdf_madd( a2, a2, d, PR ); \
5625 stbir__simdf_load1( d, d0 + dofs3 + (ofs) ); \
5626 stbir__simdf_madd( a3, a3, d, PR ); \
5627 } while(0)
5628
5629#define stbir__period1_x4_store() \
5630 do { \
5631 stbir__simdf_addhalves2( a0, a0, a1 ); \
5632 stbir__simdf_addhalves2( a2, a2, a3 ); \
5633 stbir__simdf_hadd2( a0, a0, a2 ); \
5634 stbir__simdf_store( output, a0 ); \
5635 decode_buffer = d0 + dofs3 + advance; \
5636 output += 4; \
5637 } while(0)
5638
5639#define stbir__store_output() \
5640 stbir__simdf_0123to2301( c, tot ); \
5641 stbir__simdf_add( tot, tot, c ); \
5642 stbir__simdf_0123to1230( c, tot ); \
5643 stbir__simdf_add1( tot, tot, c ); \
5644 stbir__simdf_store1( output, tot ); \
5645 stbir__next_contributor(); \
5646 output += 1;
5647
5648#else
5649
5650#define stbir__1_coeff_only() \
5651 float tot; \
5652 tot = decode[0]*hc[0];
5653
5654#define stbir__2_coeff_only() \
5655 float tot; \
5656 tot = decode[0] * hc[0]; \
5657 tot += decode[1] * hc[1];
5658
5659#define stbir__3_coeff_only() \
5660 float tot; \
5661 tot = decode[0] * hc[0]; \
5662 tot += decode[1] * hc[1]; \
5663 tot += decode[2] * hc[2];
5664
5665#define stbir__store_output_tiny() \
5666 output[0] = tot; \
5667 stbir__next_contributor(); \
5668 output += 1;
5669
5670#define stbir__4_coeff_start() \
5671 float tot0,tot1,tot2,tot3; \
5672 tot0 = decode[0] * hc[0]; \
5673 tot1 = decode[1] * hc[1]; \
5674 tot2 = decode[2] * hc[2]; \
5675 tot3 = decode[3] * hc[3];
5676
5677#define stbir__4_coeff_continue_from_4( ofs ) \
5678 tot0 += decode[0+(ofs)] * hc[0+(ofs)]; \
5679 tot1 += decode[1+(ofs)] * hc[1+(ofs)]; \
5680 tot2 += decode[2+(ofs)] * hc[2+(ofs)]; \
5681 tot3 += decode[3+(ofs)] * hc[3+(ofs)];
5682
5683#define stbir__1_coeff_remnant( ofs ) \
5684 tot0 += decode[0+(ofs)] * hc[0+(ofs)];
5685
5686#define stbir__2_coeff_remnant( ofs ) \
5687 tot0 += decode[0+(ofs)] * hc[0+(ofs)]; \
5688 tot1 += decode[1+(ofs)] * hc[1+(ofs)]; \
5689
5690#define stbir__3_coeff_remnant( ofs ) \
5691 tot0 += decode[0+(ofs)] * hc[0+(ofs)]; \
5692 tot1 += decode[1+(ofs)] * hc[1+(ofs)]; \
5693 tot2 += decode[2+(ofs)] * hc[2+(ofs)];
5694
5695#define stbir__store_output() \
5696 output[0] = (tot0+tot2)+(tot1+tot3); \
5697 stbir__next_contributor(); \
5698 output += 1;
5699
5700#endif
5701
5702#define STBIR__horizontal_channels 1
5703#define STB_IMAGE_RESIZE_DO_HORIZONTALS
5704#include STBIR__HEADER_FILENAME
5705
5706
5707//=================
5708// Do 2 channel horizontal routines
5709
5710#ifdef STBIR_SIMD
5711
5712#define stbir__1_coeff_only() \
5713 stbir__simdf tot,c,d; \
5714 STBIR_SIMD_NO_UNROLL(decode); \
5715 stbir__simdf_load1z( c, hc ); \
5716 stbir__simdf_0123to0011( c, c ); \
5717 stbir__simdf_load2( d, decode ); \
5718 stbir__simdf_mult( tot, d, c );
5719
5720#define stbir__2_coeff_only() \
5721 stbir__simdf tot,c; \
5722 STBIR_SIMD_NO_UNROLL(decode); \
5723 stbir__simdf_load2( c, hc ); \
5724 stbir__simdf_0123to0011( c, c ); \
5725 stbir__simdf_mult_mem( tot, c, decode );
5726
5727#define stbir__3_coeff_only() \
5728 stbir__simdf tot,c,cs,d; \
5729 STBIR_SIMD_NO_UNROLL(decode); \
5730 stbir__simdf_load( cs, hc ); \
5731 stbir__simdf_0123to0011( c, cs ); \
5732 stbir__simdf_mult_mem( tot, c, decode ); \
5733 stbir__simdf_0123to2222( c, cs ); \
5734 stbir__simdf_load2z( d, decode+4 ); \
5735 stbir__simdf_madd( tot, tot, d, c );
5736
5737#define stbir__store_output_tiny() \
5738 stbir__simdf_0123to2301( c, tot ); \
5739 stbir__simdf_add( tot, tot, c ); \
5740 stbir__simdf_store2( output, tot ); \
5741 stbir__next_contributor(); \
5742 output += 2;
5743
5744// polyphase period 1 for 2 channels (when have exact downsamples 2:1, 3:1, etc - common for mipmaps)
5745#define stbir__period1_setup_4() stbir__simdf P0a, P0b, cs; stbir__simdf_load( cs, hc ); stbir__simdf_0123to0011( P0a, cs ); stbir__simdf_0123to2233( P0b, cs );
5746#define stbir__period1_setup_5() stbir__simdf PR; stbir__period1_setup_4() stbir__simdf_load1z( cs, hc + 4 ); stbir__simdf_0123to0011( PR, cs );
5747#define stbir__period1_setup_8() stbir__simdf P1a, P1b; stbir__period1_setup_4() stbir__simdf_load( cs, hc + 4 ); stbir__simdf_0123to0011( P1a, cs ); stbir__simdf_0123to2233( P1b, cs );
5748#define stbir__period1_setup_9() stbir__simdf PR; stbir__period1_setup_8() stbir__simdf_load1z( cs, hc + 8 ); stbir__simdf_0123to0011( PR, cs );
5749#define stbir__period1_setup_11() stbir__simdf PR3a, PR3b; stbir__period1_setup_8() stbir__simdf_load( cs, hc + 8 ); stbir__simdf_0123to0011( PR3a, cs ); stbir__simdf_0123to2222( PR3b, cs );
5750#define stbir__period1_setup_12() stbir__simdf P2a, P2b; stbir__period1_setup_8() stbir__simdf_load( cs, hc + 8 ); stbir__simdf_0123to0011( P2a, cs ); stbir__simdf_0123to2233( P2b, cs );
5751#define stbir__period1_setup_13() stbir__simdf PR; stbir__period1_setup_12() stbir__simdf_load1z( cs, hc + 12 ); stbir__simdf_0123to0011( PR, cs );
5752#define stbir__period1_setup_16() stbir__simdf P3a, P3b; stbir__period1_setup_12() stbir__simdf_load( cs, hc + 12 ); stbir__simdf_0123to0011( P3a, cs ); stbir__simdf_0123to2233( P3b, cs );
5753#define stbir__period1_setup_17() stbir__simdf PR; stbir__period1_setup_16() stbir__simdf_load1z( cs, hc + 16 ); stbir__simdf_0123to0011( PR, cs );
5754
5755#define stbir__period1_4_coeff_start() \
5756 stbir__simdf tot0, tot1; \
5757 STBIR_SIMD_NO_UNROLL(decode); \
5758 stbir__simdf_mult_mem( tot0, P0a, decode ); \
5759 stbir__simdf_mult_mem( tot1, P0b, decode + 4 );
5760
5761#define stbir__period1_4_coeff_continue() \
5762 stbir__simdf_madd_mem( tot0, tot0, P1a, decode + 8 ); \
5763 stbir__simdf_madd_mem( tot1, tot1, P1b, decode + 12 );
5764#define stbir__period1_4_coeff_continue2() \
5765 stbir__simdf_madd_mem( tot0, tot0, P2a, decode + 16 ); \
5766 stbir__simdf_madd_mem( tot1, tot1, P2b, decode + 20 );
5767#define stbir__period1_4_coeff_continue3() \
5768 stbir__simdf_madd_mem( tot0, tot0, P3a, decode + 24 ); \
5769 stbir__simdf_madd_mem( tot1, tot1, P3b, decode + 28 );
5770
5771#define stbir__period1_1_coeff_remnant( ofs ) \
5772 { stbir__simdf d; \
5773 stbir__simdf_load2( d, decode + (ofs) * 2 ); \
5774 stbir__simdf_madd( tot0, tot0, d, PR ); }
5775#define stbir__period1_3_coeff_remnant( ofs ) \
5776 { stbir__simdf d; \
5777 stbir__simdf_madd_mem( tot0, tot0, PR3a, decode + (ofs) * 2 ); \
5778 stbir__simdf_load2z( d, decode + (ofs) * 2 + 4 ); \
5779 stbir__simdf_madd( tot1, tot1, d, PR3b ); }
5780
5781#define stbir__period1_store_output() \
5782 stbir__simdf_add( tot0, tot0, tot1 ); \
5783 stbir__simdf_0123to2301( tot1, tot0 ); \
5784 stbir__simdf_add( tot0, tot0, tot1 ); \
5785 stbir__simdf_store2( output, tot0 ); \
5786 decode_buffer += advance; \
5787 output += 2;
5788
5789#ifdef STBIR_SIMD8
5790
5791#define stbir__4_coeff_start() \
5792 stbir__simdf8 tot0,c,cs; \
5793 STBIR_SIMD_NO_UNROLL(decode); \
5794 stbir__simdf8_load4b( cs, hc ); \
5795 stbir__simdf8_0123to00112233( c, cs ); \
5796 stbir__simdf8_mult_mem( tot0, c, decode );
5797
5798#define stbir__4_coeff_continue_from_4( ofs ) \
5799 STBIR_SIMD_NO_UNROLL(decode); \
5800 stbir__simdf8_load4b( cs, hc + (ofs) ); \
5801 stbir__simdf8_0123to00112233( c, cs ); \
5802 stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*2 );
5803
5804#define stbir__1_coeff_remnant( ofs ) \
5805 { stbir__simdf t,d; \
5806 stbir__simdf_load1z( t, hc + (ofs) ); \
5807 stbir__simdf_load2( d, decode + (ofs) * 2 ); \
5808 stbir__simdf_0123to0011( t, t ); \
5809 stbir__simdf_mult( t, t, d ); \
5810 stbir__simdf8_add4( tot0, tot0, t ); }
5811
5812#define stbir__2_coeff_remnant( ofs ) \
5813 { stbir__simdf t; \
5814 stbir__simdf_load2( t, hc + (ofs) ); \
5815 stbir__simdf_0123to0011( t, t ); \
5816 stbir__simdf_mult_mem( t, t, decode+(ofs)*2 ); \
5817 stbir__simdf8_add4( tot0, tot0, t ); }
5818
5819#define stbir__3_coeff_remnant( ofs ) \
5820 { stbir__simdf8 d; \
5821 stbir__simdf8_load4b( cs, hc + (ofs) ); \
5822 stbir__simdf8_0123to00112233( c, cs ); \
5823 stbir__simdf8_load6z( d, decode+(ofs)*2 ); \
5824 stbir__simdf8_madd( tot0, tot0, c, d ); }
5825
5826#define stbir__store_output() \
5827 { stbir__simdf t,d; \
5828 stbir__simdf8_add4halves( t, stbir__if_simdf8_cast_to_simdf4(tot0), tot0 ); \
5829 stbir__simdf_0123to2301( d, t ); \
5830 stbir__simdf_add( t, t, d ); \
5831 stbir__simdf_store2( output, t ); \
5832 stbir__next_contributor(); \
5833 output += 2; }
5834
5835#else
5836
5837#define stbir__4_coeff_start() \
5838 stbir__simdf tot0,tot1,c,cs; \
5839 STBIR_SIMD_NO_UNROLL(decode); \
5840 stbir__simdf_load( cs, hc ); \
5841 stbir__simdf_0123to0011( c, cs ); \
5842 stbir__simdf_mult_mem( tot0, c, decode ); \
5843 stbir__simdf_0123to2233( c, cs ); \
5844 stbir__simdf_mult_mem( tot1, c, decode+4 );
5845
5846#define stbir__4_coeff_continue_from_4( ofs ) \
5847 STBIR_SIMD_NO_UNROLL(decode); \
5848 stbir__simdf_load( cs, hc + (ofs) ); \
5849 stbir__simdf_0123to0011( c, cs ); \
5850 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*2 ); \
5851 stbir__simdf_0123to2233( c, cs ); \
5852 stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*2+4 );
5853
5854#define stbir__1_coeff_remnant( ofs ) \
5855 { stbir__simdf d; \
5856 stbir__simdf_load1z( cs, hc + (ofs) ); \
5857 stbir__simdf_0123to0011( c, cs ); \
5858 stbir__simdf_load2( d, decode + (ofs) * 2 ); \
5859 stbir__simdf_madd( tot0, tot0, d, c ); }
5860
5861#define stbir__2_coeff_remnant( ofs ) \
5862 stbir__simdf_load2( cs, hc + (ofs) ); \
5863 stbir__simdf_0123to0011( c, cs ); \
5864 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*2 );
5865
5866#define stbir__3_coeff_remnant( ofs ) \
5867 { stbir__simdf d; \
5868 stbir__simdf_load( cs, hc + (ofs) ); \
5869 stbir__simdf_0123to0011( c, cs ); \
5870 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*2 ); \
5871 stbir__simdf_0123to2222( c, cs ); \
5872 stbir__simdf_load2z( d, decode + (ofs) * 2 + 4 ); \
5873 stbir__simdf_madd( tot1, tot1, d, c ); }
5874
5875#define stbir__store_output() \
5876 stbir__simdf_add( tot0, tot0, tot1 ); \
5877 stbir__simdf_0123to2301( c, tot0 ); \
5878 stbir__simdf_add( tot0, tot0, c ); \
5879 stbir__simdf_store2( output, tot0 ); \
5880 stbir__next_contributor(); \
5881 output += 2;
5882
5883#endif
5884
5885#else
5886
5887#define stbir__1_coeff_only() \
5888 float tota,totb,c; \
5889 c = hc[0]; \
5890 tota = decode[0]*c; \
5891 totb = decode[1]*c;
5892
5893#define stbir__2_coeff_only() \
5894 float tota,totb,c; \
5895 c = hc[0]; \
5896 tota = decode[0]*c; \
5897 totb = decode[1]*c; \
5898 c = hc[1]; \
5899 tota += decode[2]*c; \
5900 totb += decode[3]*c;
5901
5902// this weird order of add matches the simd
5903#define stbir__3_coeff_only() \
5904 float tota,totb,c; \
5905 c = hc[0]; \
5906 tota = decode[0]*c; \
5907 totb = decode[1]*c; \
5908 c = hc[2]; \
5909 tota += decode[4]*c; \
5910 totb += decode[5]*c; \
5911 c = hc[1]; \
5912 tota += decode[2]*c; \
5913 totb += decode[3]*c;
5914
5915#define stbir__store_output_tiny() \
5916 output[0] = tota; \
5917 output[1] = totb; \
5918 stbir__next_contributor(); \
5919 output += 2;
5920
5921#define stbir__4_coeff_start() \
5922 float tota0,tota1,tota2,tota3,totb0,totb1,totb2,totb3,c; \
5923 c = hc[0]; \
5924 tota0 = decode[0]*c; \
5925 totb0 = decode[1]*c; \
5926 c = hc[1]; \
5927 tota1 = decode[2]*c; \
5928 totb1 = decode[3]*c; \
5929 c = hc[2]; \
5930 tota2 = decode[4]*c; \
5931 totb2 = decode[5]*c; \
5932 c = hc[3]; \
5933 tota3 = decode[6]*c; \
5934 totb3 = decode[7]*c;
5935
5936#define stbir__4_coeff_continue_from_4( ofs ) \
5937 c = hc[0+(ofs)]; \
5938 tota0 += decode[0+(ofs)*2]*c; \
5939 totb0 += decode[1+(ofs)*2]*c; \
5940 c = hc[1+(ofs)]; \
5941 tota1 += decode[2+(ofs)*2]*c; \
5942 totb1 += decode[3+(ofs)*2]*c; \
5943 c = hc[2+(ofs)]; \
5944 tota2 += decode[4+(ofs)*2]*c; \
5945 totb2 += decode[5+(ofs)*2]*c; \
5946 c = hc[3+(ofs)]; \
5947 tota3 += decode[6+(ofs)*2]*c; \
5948 totb3 += decode[7+(ofs)*2]*c;
5949
5950#define stbir__1_coeff_remnant( ofs ) \
5951 c = hc[0+(ofs)]; \
5952 tota0 += decode[0+(ofs)*2] * c; \
5953 totb0 += decode[1+(ofs)*2] * c;
5954
5955#define stbir__2_coeff_remnant( ofs ) \
5956 c = hc[0+(ofs)]; \
5957 tota0 += decode[0+(ofs)*2] * c; \
5958 totb0 += decode[1+(ofs)*2] * c; \
5959 c = hc[1+(ofs)]; \
5960 tota1 += decode[2+(ofs)*2] * c; \
5961 totb1 += decode[3+(ofs)*2] * c;
5962
5963#define stbir__3_coeff_remnant( ofs ) \
5964 c = hc[0+(ofs)]; \
5965 tota0 += decode[0+(ofs)*2] * c; \
5966 totb0 += decode[1+(ofs)*2] * c; \
5967 c = hc[1+(ofs)]; \
5968 tota1 += decode[2+(ofs)*2] * c; \
5969 totb1 += decode[3+(ofs)*2] * c; \
5970 c = hc[2+(ofs)]; \
5971 tota2 += decode[4+(ofs)*2] * c; \
5972 totb2 += decode[5+(ofs)*2] * c;
5973
5974#define stbir__store_output() \
5975 output[0] = (tota0+tota2)+(tota1+tota3); \
5976 output[1] = (totb0+totb2)+(totb1+totb3); \
5977 stbir__next_contributor(); \
5978 output += 2;
5979
5980#endif
5981
5982#define STBIR__horizontal_channels 2
5983#define STB_IMAGE_RESIZE_DO_HORIZONTALS
5984#include STBIR__HEADER_FILENAME
5985
5986
5987//=================
5988// Do 3 channel horizontal routines
5989
5990#ifdef STBIR_SIMD
5991
5992#define stbir__1_coeff_only() \
5993 stbir__simdf tot,c,d; \
5994 STBIR_SIMD_NO_UNROLL(decode); \
5995 stbir__simdf_load1z( c, hc ); \
5996 stbir__simdf_0123to0001( c, c ); \
5997 stbir__simdf_load( d, decode ); \
5998 stbir__simdf_mult( tot, d, c );
5999
6000#define stbir__2_coeff_only() \
6001 stbir__simdf tot,c,cs,d; \
6002 STBIR_SIMD_NO_UNROLL(decode); \
6003 stbir__simdf_load2( cs, hc ); \
6004 stbir__simdf_0123to0000( c, cs ); \
6005 stbir__simdf_load( d, decode ); \
6006 stbir__simdf_mult( tot, d, c ); \
6007 stbir__simdf_0123to1111( c, cs ); \
6008 stbir__simdf_load( d, decode+3 ); \
6009 stbir__simdf_madd( tot, tot, d, c );
6010
6011#define stbir__3_coeff_only() \
6012 stbir__simdf tot,c,d,cs; \
6013 STBIR_SIMD_NO_UNROLL(decode); \
6014 stbir__simdf_load( cs, hc ); \
6015 stbir__simdf_0123to0000( c, cs ); \
6016 stbir__simdf_load( d, decode ); \
6017 stbir__simdf_mult( tot, d, c ); \
6018 stbir__simdf_0123to1111( c, cs ); \
6019 stbir__simdf_load( d, decode+3 ); \
6020 stbir__simdf_madd( tot, tot, d, c ); \
6021 stbir__simdf_0123to2222( c, cs ); \
6022 stbir__simdf_load( d, decode+6 ); \
6023 stbir__simdf_madd( tot, tot, d, c );
6024
6025#define stbir__store_output_tiny() \
6026 stbir__simdf_store2( output, tot ); \
6027 stbir__simdf_0123to2301( tot, tot ); \
6028 stbir__simdf_store1( output+2, tot ); \
6029 stbir__next_contributor(); \
6030 output += 3;
6031
6032#ifdef STBIR_SIMD8
6033
6034// we're loading from the XXXYYY decode by -1 to get the XXXYYY into different halves of the AVX reg fyi
6035#define stbir__4_coeff_start() \
6036 stbir__simdf8 tot0,tot1,c,cs; stbir__simdf t; \
6037 STBIR_SIMD_NO_UNROLL(decode); \
6038 stbir__simdf8_load4b( cs, hc ); \
6039 stbir__simdf8_0123to00001111( c, cs ); \
6040 stbir__simdf8_mult_mem( tot0, c, decode - 1 ); \
6041 stbir__simdf8_0123to22223333( c, cs ); \
6042 stbir__simdf8_mult_mem( tot1, c, decode+6 - 1 );
6043
6044#define stbir__4_coeff_continue_from_4( ofs ) \
6045 STBIR_SIMD_NO_UNROLL(decode); \
6046 stbir__simdf8_load4b( cs, hc + (ofs) ); \
6047 stbir__simdf8_0123to00001111( c, cs ); \
6048 stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*3 - 1 ); \
6049 stbir__simdf8_0123to22223333( c, cs ); \
6050 stbir__simdf8_madd_mem( tot1, tot1, c, decode+(ofs)*3 + 6 - 1 );
6051
6052#define stbir__1_coeff_remnant( ofs ) \
6053 STBIR_SIMD_NO_UNROLL(decode); \
6054 stbir__simdf_load1rep4( t, hc + (ofs) ); \
6055 stbir__simdf8_madd_mem4( tot0, tot0, t, decode+(ofs)*3 - 1 );
6056
6057#define stbir__2_coeff_remnant( ofs ) \
6058 STBIR_SIMD_NO_UNROLL(decode); \
6059 stbir__simdf8_load4b( cs, hc + (ofs) - 2 ); \
6060 stbir__simdf8_0123to22223333( c, cs ); \
6061 stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*3 - 1 );
6062
6063 #define stbir__3_coeff_remnant( ofs ) \
6064 STBIR_SIMD_NO_UNROLL(decode); \
6065 stbir__simdf8_load4b( cs, hc + (ofs) ); \
6066 stbir__simdf8_0123to00001111( c, cs ); \
6067 stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*3 - 1 ); \
6068 stbir__simdf8_0123to2222( t, cs ); \
6069 stbir__simdf8_madd_mem4( tot1, tot1, t, decode+(ofs)*3 + 6 - 1 );
6070
6071// polyphase period 1 for 3 channels (when have exact downsamples 2:1, 3:1, etc - common for mipmaps)
6072#define stbir__period1_setup_4() stbir__simdf8 P01, P23, cs; stbir__simdf8_load4b( cs, hc ); stbir__simdf8_0123to00001111( P01, cs ); stbir__simdf8_0123to22223333( P23, cs );
6073#define stbir__period1_setup_5() stbir__simdf PR; stbir__period1_setup_4() stbir__simdf_load1rep4( PR, hc + 4 );
6074#define stbir__period1_setup_8() stbir__simdf8 P45, P67; stbir__period1_setup_4() stbir__simdf8_load4b( cs, hc + 4 ); stbir__simdf8_0123to00001111( P45, cs ); stbir__simdf8_0123to22223333( P67, cs );
6075#define stbir__period1_setup_9() stbir__simdf PR; stbir__period1_setup_8() stbir__simdf_load1rep4( PR, hc + 8 );
6076#define stbir__period1_setup_11() stbir__simdf8 P89; stbir__simdf PR; stbir__period1_setup_8() stbir__simdf8_load4b( cs, hc + 8 ); stbir__simdf8_0123to00001111( P89, cs ); stbir__simdf8_0123to2222( PR, cs );
6077#define stbir__period1_setup_12() stbir__simdf8 P89, PAB; stbir__period1_setup_8() stbir__simdf8_load4b( cs, hc + 8 ); stbir__simdf8_0123to00001111( P89, cs ); stbir__simdf8_0123to22223333( PAB, cs );
6078#define stbir__period1_setup_13() stbir__simdf PR; stbir__period1_setup_12() stbir__simdf_load1rep4( PR, hc + 12 );
6079#define stbir__period1_setup_16() stbir__simdf8 PCD, PEF; stbir__period1_setup_12() stbir__simdf8_load4b( cs, hc + 12 ); stbir__simdf8_0123to00001111( PCD, cs ); stbir__simdf8_0123to22223333( PEF, cs );
6080#define stbir__period1_setup_17() stbir__simdf PR; stbir__period1_setup_16() stbir__simdf_load1rep4( PR, hc + 16 );
6081
6082#define stbir__period1_4_coeff_start() \
6083 stbir__simdf8 tot0, tot1; stbir__simdf t; \
6084 STBIR_SIMD_NO_UNROLL(decode); \
6085 stbir__simdf8_mult_mem( tot0, P01, decode - 1 ); \
6086 stbir__simdf8_mult_mem( tot1, P23, decode + 6 - 1 );
6087
6088#define stbir__period1_4_coeff_continue() \
6089 stbir__simdf8_madd_mem( tot0, tot0, P45, decode + 12 - 1 ); \
6090 stbir__simdf8_madd_mem( tot1, tot1, P67, decode + 18 - 1 );
6091#define stbir__period1_4_coeff_continue2() \
6092 stbir__simdf8_madd_mem( tot0, tot0, P89, decode + 24 - 1 ); \
6093 stbir__simdf8_madd_mem( tot1, tot1, PAB, decode + 30 - 1 );
6094#define stbir__period1_4_coeff_continue3() \
6095 stbir__simdf8_madd_mem( tot0, tot0, PCD, decode + 36 - 1 ); \
6096 stbir__simdf8_madd_mem( tot1, tot1, PEF, decode + 42 - 1 );
6097
6098#define stbir__period1_1_coeff_remnant( ofs ) \
6099 stbir__simdf8_madd_mem4( tot0, tot0, PR, decode + (ofs) * 3 - 1 );
6100#define stbir__period1_3_coeff_remnant( ofs ) \
6101 stbir__simdf8_madd_mem( tot0, tot0, P89, decode + (ofs) * 3 - 1 ); \
6102 stbir__simdf8_madd_mem4( tot1, tot1, PR, decode + (ofs) * 3 + 6 - 1 );
6103
6104#define stbir__period1_store_output() \
6105 stbir__simdf8_add( tot0, tot0, tot1 ); \
6106 stbir__simdf_0123to1230( t, stbir__if_simdf8_cast_to_simdf4( tot0 ) ); \
6107 stbir__simdf8_add4halves( t, t, tot0 ); \
6108 decode_buffer += advance; \
6109 output += 3; \
6110 if ( output < output_end ) \
6111 { \
6112 stbir__simdf_store( output-3, t ); \
6113 continue; \
6114 } \
6115 { stbir__simdf tt; stbir__simdf_0123to2301( tt, t ); \
6116 stbir__simdf_store2( output-3, t ); \
6117 stbir__simdf_store1( output+2-3, tt ); } \
6118 break;
6119
6120#define stbir__store_output() \
6121 stbir__simdf8_add( tot0, tot0, tot1 ); \
6122 stbir__simdf_0123to1230( t, stbir__if_simdf8_cast_to_simdf4( tot0 ) ); \
6123 stbir__simdf8_add4halves( t, t, tot0 ); \
6124 stbir__next_contributor(); \
6125 output += 3; \
6126 if ( output < output_end ) \
6127 { \
6128 stbir__simdf_store( output-3, t ); \
6129 continue; \
6130 } \
6131 { stbir__simdf tt; stbir__simdf_0123to2301( tt, t ); \
6132 stbir__simdf_store2( output-3, t ); \
6133 stbir__simdf_store1( output+2-3, tt ); } \
6134 break;
6135
6136
6137#else
6138
6139#define stbir__4_coeff_start() \
6140 stbir__simdf tot0,tot1,tot2,c,cs; \
6141 STBIR_SIMD_NO_UNROLL(decode); \
6142 stbir__simdf_load( cs, hc ); \
6143 stbir__simdf_0123to0001( c, cs ); \
6144 stbir__simdf_mult_mem( tot0, c, decode ); \
6145 stbir__simdf_0123to1122( c, cs ); \
6146 stbir__simdf_mult_mem( tot1, c, decode+4 ); \
6147 stbir__simdf_0123to2333( c, cs ); \
6148 stbir__simdf_mult_mem( tot2, c, decode+8 );
6149
6150#define stbir__4_coeff_continue_from_4( ofs ) \
6151 STBIR_SIMD_NO_UNROLL(decode); \
6152 stbir__simdf_load( cs, hc + (ofs) ); \
6153 stbir__simdf_0123to0001( c, cs ); \
6154 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*3 ); \
6155 stbir__simdf_0123to1122( c, cs ); \
6156 stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*3+4 ); \
6157 stbir__simdf_0123to2333( c, cs ); \
6158 stbir__simdf_madd_mem( tot2, tot2, c, decode+(ofs)*3+8 );
6159
6160#define stbir__1_coeff_remnant( ofs ) \
6161 STBIR_SIMD_NO_UNROLL(decode); \
6162 stbir__simdf_load1z( c, hc + (ofs) ); \
6163 stbir__simdf_0123to0001( c, c ); \
6164 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*3 );
6165
6166#define stbir__2_coeff_remnant( ofs ) \
6167 { stbir__simdf d; \
6168 STBIR_SIMD_NO_UNROLL(decode); \
6169 stbir__simdf_load2z( cs, hc + (ofs) ); \
6170 stbir__simdf_0123to0001( c, cs ); \
6171 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*3 ); \
6172 stbir__simdf_0123to1122( c, cs ); \
6173 stbir__simdf_load2z( d, decode+(ofs)*3+4 ); \
6174 stbir__simdf_madd( tot1, tot1, c, d ); }
6175
6176#define stbir__3_coeff_remnant( ofs ) \
6177 { stbir__simdf d; \
6178 STBIR_SIMD_NO_UNROLL(decode); \
6179 stbir__simdf_load( cs, hc + (ofs) ); \
6180 stbir__simdf_0123to0001( c, cs ); \
6181 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*3 ); \
6182 stbir__simdf_0123to1122( c, cs ); \
6183 stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*3+4 ); \
6184 stbir__simdf_0123to2222( c, cs ); \
6185 stbir__simdf_load1z( d, decode+(ofs)*3+8 ); \
6186 stbir__simdf_madd( tot2, tot2, c, d ); }
6187
6188// polyphase period 1 for 3 channels (when have exact downsamples 2:1, 3:1, etc - common for mipmaps)
6189#define stbir__period1_setup_4() stbir__simdf P0a, P0b, P0c, cs; stbir__simdf_load( cs, hc ); stbir__simdf_0123to0001( P0a, cs ); stbir__simdf_0123to1122( P0b, cs ); stbir__simdf_0123to2333( P0c, cs );
6190#define stbir__period1_setup_5() stbir__simdf PR; stbir__period1_setup_4() stbir__simdf_load1z( cs, hc + 4 ); stbir__simdf_0123to0001( PR, cs );
6191#define stbir__period1_setup_8() stbir__simdf P1a, P1b, P1c; stbir__period1_setup_4() stbir__simdf_load( cs, hc + 4 ); stbir__simdf_0123to0001( P1a, cs ); stbir__simdf_0123to1122( P1b, cs ); stbir__simdf_0123to2333( P1c, cs );
6192#define stbir__period1_setup_9() stbir__simdf PR; stbir__period1_setup_8() stbir__simdf_load1z( cs, hc + 8 ); stbir__simdf_0123to0001( PR, cs );
6193#define stbir__period1_setup_11() stbir__simdf PR3a, PR3b, PR3c; stbir__period1_setup_8() stbir__simdf_load( cs, hc + 8 ); stbir__simdf_0123to0001( PR3a, cs ); stbir__simdf_0123to1122( PR3b, cs ); stbir__simdf_0123to2222( PR3c, cs );
6194#define stbir__period1_setup_12() stbir__simdf P2a, P2b, P2c; stbir__period1_setup_8() stbir__simdf_load( cs, hc + 8 ); stbir__simdf_0123to0001( P2a, cs ); stbir__simdf_0123to1122( P2b, cs ); stbir__simdf_0123to2333( P2c, cs );
6195#define stbir__period1_setup_13() stbir__simdf PR; stbir__period1_setup_12() stbir__simdf_load1z( cs, hc + 12 ); stbir__simdf_0123to0001( PR, cs );
6196#define stbir__period1_setup_16() stbir__simdf P3a, P3b, P3c; stbir__period1_setup_12() stbir__simdf_load( cs, hc + 12 ); stbir__simdf_0123to0001( P3a, cs ); stbir__simdf_0123to1122( P3b, cs ); stbir__simdf_0123to2333( P3c, cs );
6197#define stbir__period1_setup_17() stbir__simdf PR; stbir__period1_setup_16() stbir__simdf_load1z( cs, hc + 16 ); stbir__simdf_0123to0001( PR, cs );
6198
6199#define stbir__period1_4_coeff_start() \
6200 stbir__simdf tot0, tot1, tot2, c; \
6201 STBIR_SIMD_NO_UNROLL(decode); \
6202 stbir__simdf_mult_mem( tot0, P0a, decode ); \
6203 stbir__simdf_mult_mem( tot1, P0b, decode + 4 ); \
6204 stbir__simdf_mult_mem( tot2, P0c, decode + 8 );
6205
6206#define stbir__period1_4_coeff_continue() \
6207 stbir__simdf_madd_mem( tot0, tot0, P1a, decode + 12 ); \
6208 stbir__simdf_madd_mem( tot1, tot1, P1b, decode + 16 ); \
6209 stbir__simdf_madd_mem( tot2, tot2, P1c, decode + 20 );
6210#define stbir__period1_4_coeff_continue2() \
6211 stbir__simdf_madd_mem( tot0, tot0, P2a, decode + 24 ); \
6212 stbir__simdf_madd_mem( tot1, tot1, P2b, decode + 28 ); \
6213 stbir__simdf_madd_mem( tot2, tot2, P2c, decode + 32 );
6214#define stbir__period1_4_coeff_continue3() \
6215 stbir__simdf_madd_mem( tot0, tot0, P3a, decode + 36 ); \
6216 stbir__simdf_madd_mem( tot1, tot1, P3b, decode + 40 ); \
6217 stbir__simdf_madd_mem( tot2, tot2, P3c, decode + 44 );
6218
6219#define stbir__period1_1_coeff_remnant( ofs ) \
6220 stbir__simdf_madd_mem( tot0, tot0, PR, decode + (ofs) * 3 );
6221#define stbir__period1_3_coeff_remnant( ofs ) \
6222 { stbir__simdf d; \
6223 stbir__simdf_madd_mem( tot0, tot0, PR3a, decode + (ofs) * 3 ); \
6224 stbir__simdf_madd_mem( tot1, tot1, PR3b, decode + (ofs) * 3 + 4 ); \
6225 stbir__simdf_load1z( d, decode + (ofs) * 3 + 8 ); \
6226 stbir__simdf_madd( tot2, tot2, PR3c, d ); }
6227
6228#define stbir__period1_store_output() \
6229 stbir__simdf_0123ABCDto3ABx( c, tot0, tot1 ); \
6230 stbir__simdf_0123ABCDto23Ax( cs, tot1, tot2 ); \
6231 stbir__simdf_0123to1230( tot2, tot2 ); \
6232 stbir__simdf_add( tot0, tot0, cs ); \
6233 stbir__simdf_add( c, c, tot2 ); \
6234 stbir__simdf_add( tot0, tot0, c ); \
6235 decode_buffer += advance; \
6236 output += 3; \
6237 if ( output < output_end ) \
6238 { \
6239 stbir__simdf_store( output-3, tot0 ); \
6240 continue; \
6241 } \
6242 stbir__simdf_0123to2301( tot1, tot0 ); \
6243 stbir__simdf_store2( output-3, tot0 ); \
6244 stbir__simdf_store1( output+2-3, tot1 ); \
6245 break;
6246
6247#define stbir__store_output() \
6248 stbir__simdf_0123ABCDto3ABx( c, tot0, tot1 ); \
6249 stbir__simdf_0123ABCDto23Ax( cs, tot1, tot2 ); \
6250 stbir__simdf_0123to1230( tot2, tot2 ); \
6251 stbir__simdf_add( tot0, tot0, cs ); \
6252 stbir__simdf_add( c, c, tot2 ); \
6253 stbir__simdf_add( tot0, tot0, c ); \
6254 stbir__next_contributor(); \
6255 output += 3; \
6256 if ( output < output_end ) \
6257 { \
6258 stbir__simdf_store( output-3, tot0 ); \
6259 continue; \
6260 } \
6261 stbir__simdf_0123to2301( tot1, tot0 ); \
6262 stbir__simdf_store2( output-3, tot0 ); \
6263 stbir__simdf_store1( output+2-3, tot1 ); \
6264 break;
6265
6266#endif
6267
6268#else
6269
6270#define stbir__1_coeff_only() \
6271 float tot0, tot1, tot2, c; \
6272 c = hc[0]; \
6273 tot0 = decode[0]*c; \
6274 tot1 = decode[1]*c; \
6275 tot2 = decode[2]*c;
6276
6277#define stbir__2_coeff_only() \
6278 float tot0, tot1, tot2, c; \
6279 c = hc[0]; \
6280 tot0 = decode[0]*c; \
6281 tot1 = decode[1]*c; \
6282 tot2 = decode[2]*c; \
6283 c = hc[1]; \
6284 tot0 += decode[3]*c; \
6285 tot1 += decode[4]*c; \
6286 tot2 += decode[5]*c;
6287
6288#define stbir__3_coeff_only() \
6289 float tot0, tot1, tot2, c; \
6290 c = hc[0]; \
6291 tot0 = decode[0]*c; \
6292 tot1 = decode[1]*c; \
6293 tot2 = decode[2]*c; \
6294 c = hc[1]; \
6295 tot0 += decode[3]*c; \
6296 tot1 += decode[4]*c; \
6297 tot2 += decode[5]*c; \
6298 c = hc[2]; \
6299 tot0 += decode[6]*c; \
6300 tot1 += decode[7]*c; \
6301 tot2 += decode[8]*c;
6302
6303#define stbir__store_output_tiny() \
6304 output[0] = tot0; \
6305 output[1] = tot1; \
6306 output[2] = tot2; \
6307 stbir__next_contributor(); \
6308 output += 3;
6309
6310#define stbir__4_coeff_start() \
6311 float tota0,tota1,tota2,totb0,totb1,totb2,totc0,totc1,totc2,totd0,totd1,totd2,c; \
6312 c = hc[0]; \
6313 tota0 = decode[0]*c; \
6314 tota1 = decode[1]*c; \
6315 tota2 = decode[2]*c; \
6316 c = hc[1]; \
6317 totb0 = decode[3]*c; \
6318 totb1 = decode[4]*c; \
6319 totb2 = decode[5]*c; \
6320 c = hc[2]; \
6321 totc0 = decode[6]*c; \
6322 totc1 = decode[7]*c; \
6323 totc2 = decode[8]*c; \
6324 c = hc[3]; \
6325 totd0 = decode[9]*c; \
6326 totd1 = decode[10]*c; \
6327 totd2 = decode[11]*c;
6328
6329#define stbir__4_coeff_continue_from_4( ofs ) \
6330 c = hc[0+(ofs)]; \
6331 tota0 += decode[0+(ofs)*3]*c; \
6332 tota1 += decode[1+(ofs)*3]*c; \
6333 tota2 += decode[2+(ofs)*3]*c; \
6334 c = hc[1+(ofs)]; \
6335 totb0 += decode[3+(ofs)*3]*c; \
6336 totb1 += decode[4+(ofs)*3]*c; \
6337 totb2 += decode[5+(ofs)*3]*c; \
6338 c = hc[2+(ofs)]; \
6339 totc0 += decode[6+(ofs)*3]*c; \
6340 totc1 += decode[7+(ofs)*3]*c; \
6341 totc2 += decode[8+(ofs)*3]*c; \
6342 c = hc[3+(ofs)]; \
6343 totd0 += decode[9+(ofs)*3]*c; \
6344 totd1 += decode[10+(ofs)*3]*c; \
6345 totd2 += decode[11+(ofs)*3]*c;
6346
6347#define stbir__1_coeff_remnant( ofs ) \
6348 c = hc[0+(ofs)]; \
6349 tota0 += decode[0+(ofs)*3]*c; \
6350 tota1 += decode[1+(ofs)*3]*c; \
6351 tota2 += decode[2+(ofs)*3]*c;
6352
6353#define stbir__2_coeff_remnant( ofs ) \
6354 c = hc[0+(ofs)]; \
6355 tota0 += decode[0+(ofs)*3]*c; \
6356 tota1 += decode[1+(ofs)*3]*c; \
6357 tota2 += decode[2+(ofs)*3]*c; \
6358 c = hc[1+(ofs)]; \
6359 totb0 += decode[3+(ofs)*3]*c; \
6360 totb1 += decode[4+(ofs)*3]*c; \
6361 totb2 += decode[5+(ofs)*3]*c; \
6362
6363#define stbir__3_coeff_remnant( ofs ) \
6364 c = hc[0+(ofs)]; \
6365 tota0 += decode[0+(ofs)*3]*c; \
6366 tota1 += decode[1+(ofs)*3]*c; \
6367 tota2 += decode[2+(ofs)*3]*c; \
6368 c = hc[1+(ofs)]; \
6369 totb0 += decode[3+(ofs)*3]*c; \
6370 totb1 += decode[4+(ofs)*3]*c; \
6371 totb2 += decode[5+(ofs)*3]*c; \
6372 c = hc[2+(ofs)]; \
6373 totc0 += decode[6+(ofs)*3]*c; \
6374 totc1 += decode[7+(ofs)*3]*c; \
6375 totc2 += decode[8+(ofs)*3]*c;
6376
6377#define stbir__store_output() \
6378 output[0] = (tota0+totc0)+(totb0+totd0); \
6379 output[1] = (tota1+totc1)+(totb1+totd1); \
6380 output[2] = (tota2+totc2)+(totb2+totd2); \
6381 stbir__next_contributor(); \
6382 output += 3;
6383
6384#endif
6385
6386#define STBIR__horizontal_channels 3
6387#define STB_IMAGE_RESIZE_DO_HORIZONTALS
6388#include STBIR__HEADER_FILENAME
6389
6390//=================
6391// Do 4 channel horizontal routines
6392
6393#ifdef STBIR_SIMD
6394
6395#define stbir__1_coeff_only() \
6396 stbir__simdf tot,c; \
6397 STBIR_SIMD_NO_UNROLL(decode); \
6398 stbir__simdf_load1( c, hc ); \
6399 stbir__simdf_0123to0000( c, c ); \
6400 stbir__simdf_mult_mem( tot, c, decode );
6401
6402#define stbir__2_coeff_only() \
6403 stbir__simdf tot,c,cs; \
6404 STBIR_SIMD_NO_UNROLL(decode); \
6405 stbir__simdf_load2( cs, hc ); \
6406 stbir__simdf_0123to0000( c, cs ); \
6407 stbir__simdf_mult_mem( tot, c, decode ); \
6408 stbir__simdf_0123to1111( c, cs ); \
6409 stbir__simdf_madd_mem( tot, tot, c, decode+4 );
6410
6411#define stbir__3_coeff_only() \
6412 stbir__simdf tot,c,cs; \
6413 STBIR_SIMD_NO_UNROLL(decode); \
6414 stbir__simdf_load( cs, hc ); \
6415 stbir__simdf_0123to0000( c, cs ); \
6416 stbir__simdf_mult_mem( tot, c, decode ); \
6417 stbir__simdf_0123to1111( c, cs ); \
6418 stbir__simdf_madd_mem( tot, tot, c, decode+4 ); \
6419 stbir__simdf_0123to2222( c, cs ); \
6420 stbir__simdf_madd_mem( tot, tot, c, decode+8 );
6421
6422#define stbir__store_output_tiny() \
6423 stbir__simdf_store( output, tot ); \
6424 stbir__next_contributor(); \
6425 output += 4;
6426
6427#ifdef STBIR_SIMD8
6428
6429#define stbir__4_coeff_start() \
6430 stbir__simdf8 tot0,c,cs; stbir__simdf t; \
6431 STBIR_SIMD_NO_UNROLL(decode); \
6432 stbir__simdf8_load4b( cs, hc ); \
6433 stbir__simdf8_0123to00001111( c, cs ); \
6434 stbir__simdf8_mult_mem( tot0, c, decode ); \
6435 stbir__simdf8_0123to22223333( c, cs ); \
6436 stbir__simdf8_madd_mem( tot0, tot0, c, decode+8 );
6437
6438#define stbir__4_coeff_continue_from_4( ofs ) \
6439 STBIR_SIMD_NO_UNROLL(decode); \
6440 stbir__simdf8_load4b( cs, hc + (ofs) ); \
6441 stbir__simdf8_0123to00001111( c, cs ); \
6442 stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*4 ); \
6443 stbir__simdf8_0123to22223333( c, cs ); \
6444 stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*4+8 );
6445
6446#define stbir__1_coeff_remnant( ofs ) \
6447 STBIR_SIMD_NO_UNROLL(decode); \
6448 stbir__simdf_load1rep4( t, hc + (ofs) ); \
6449 stbir__simdf8_madd_mem4( tot0, tot0, t, decode+(ofs)*4 );
6450
6451#define stbir__2_coeff_remnant( ofs ) \
6452 STBIR_SIMD_NO_UNROLL(decode); \
6453 stbir__simdf8_load4b( cs, hc + (ofs) - 2 ); \
6454 stbir__simdf8_0123to22223333( c, cs ); \
6455 stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*4 );
6456
6457 #define stbir__3_coeff_remnant( ofs ) \
6458 STBIR_SIMD_NO_UNROLL(decode); \
6459 stbir__simdf8_load4b( cs, hc + (ofs) ); \
6460 stbir__simdf8_0123to00001111( c, cs ); \
6461 stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*4 ); \
6462 stbir__simdf8_0123to2222( t, cs ); \
6463 stbir__simdf8_madd_mem4( tot0, tot0, t, decode+(ofs)*4+8 );
6464
6465// polyphase period 1 for 4 channels (when have exact downsamples 2:1, 3:1, etc - common for mipmaps)
6466#define stbir__period1_setup_4() stbir__simdf8 P01, P23, cs; stbir__simdf8_load4b( cs, hc ); stbir__simdf8_0123to00001111( P01, cs ); stbir__simdf8_0123to22223333( P23, cs );
6467#define stbir__period1_setup_5() stbir__simdf PR; stbir__period1_setup_4() stbir__simdf_load1rep4( PR, hc + 4 );
6468#define stbir__period1_setup_8() stbir__simdf8 P45, P67; stbir__period1_setup_4() stbir__simdf8_load4b( cs, hc + 4 ); stbir__simdf8_0123to00001111( P45, cs ); stbir__simdf8_0123to22223333( P67, cs );
6469#define stbir__period1_setup_9() stbir__simdf PR; stbir__period1_setup_8() stbir__simdf_load1rep4( PR, hc + 8 );
6470#define stbir__period1_setup_11() stbir__simdf8 P89; stbir__simdf PR; stbir__period1_setup_8() stbir__simdf8_load4b( cs, hc + 8 ); stbir__simdf8_0123to00001111( P89, cs ); stbir__simdf8_0123to2222( PR, cs );
6471#define stbir__period1_setup_12() stbir__simdf8 P89, PAB; stbir__period1_setup_8() stbir__simdf8_load4b( cs, hc + 8 ); stbir__simdf8_0123to00001111( P89, cs ); stbir__simdf8_0123to22223333( PAB, cs );
6472#define stbir__period1_setup_13() stbir__simdf PR; stbir__period1_setup_12() stbir__simdf_load1rep4( PR, hc + 12 );
6473#define stbir__period1_setup_16() stbir__simdf8 PCD, PEF; stbir__period1_setup_12() stbir__simdf8_load4b( cs, hc + 12 ); stbir__simdf8_0123to00001111( PCD, cs ); stbir__simdf8_0123to22223333( PEF, cs );
6474#define stbir__period1_setup_17() stbir__simdf PR; stbir__period1_setup_16() stbir__simdf_load1rep4( PR, hc + 16 );
6475
6476#define stbir__period1_4_coeff_start() \
6477 stbir__simdf8 tot0; stbir__simdf t; \
6478 STBIR_SIMD_NO_UNROLL(decode); \
6479 stbir__simdf8_mult_mem( tot0, P01, decode ); \
6480 stbir__simdf8_madd_mem( tot0, tot0, P23, decode + 8 );
6481
6482#define stbir__period1_4_coeff_continue() \
6483 stbir__simdf8_madd_mem( tot0, tot0, P45, decode + 16 ); \
6484 stbir__simdf8_madd_mem( tot0, tot0, P67, decode + 24 );
6485#define stbir__period1_4_coeff_continue2() \
6486 stbir__simdf8_madd_mem( tot0, tot0, P89, decode + 32 ); \
6487 stbir__simdf8_madd_mem( tot0, tot0, PAB, decode + 40 );
6488#define stbir__period1_4_coeff_continue3() \
6489 stbir__simdf8_madd_mem( tot0, tot0, PCD, decode + 48 ); \
6490 stbir__simdf8_madd_mem( tot0, tot0, PEF, decode + 56 );
6491
6492#define stbir__period1_1_coeff_remnant( ofs ) \
6493 stbir__simdf8_madd_mem4( tot0, tot0, PR, decode + (ofs) * 4 );
6494#define stbir__period1_3_coeff_remnant( ofs ) \
6495 stbir__simdf8_madd_mem( tot0, tot0, P89, decode + (ofs) * 4 ); \
6496 stbir__simdf8_madd_mem4( tot0, tot0, PR, decode + (ofs) * 4 + 8 );
6497
6498#define stbir__period1_store_output() \
6499 stbir__simdf8_add4halves( t, stbir__if_simdf8_cast_to_simdf4(tot0), tot0 ); \
6500 stbir__simdf_store( output, t ); \
6501 decode_buffer += advance; \
6502 output += 4;
6503
6504#define stbir__store_output() \
6505 stbir__simdf8_add4halves( t, stbir__if_simdf8_cast_to_simdf4(tot0), tot0 ); \
6506 stbir__simdf_store( output, t ); \
6507 stbir__next_contributor(); \
6508 output += 4;
6509
6510#else
6511
6512#define stbir__4_coeff_start() \
6513 stbir__simdf tot0,tot1,c,cs; \
6514 STBIR_SIMD_NO_UNROLL(decode); \
6515 stbir__simdf_load( cs, hc ); \
6516 stbir__simdf_0123to0000( c, cs ); \
6517 stbir__simdf_mult_mem( tot0, c, decode ); \
6518 stbir__simdf_0123to1111( c, cs ); \
6519 stbir__simdf_mult_mem( tot1, c, decode+4 ); \
6520 stbir__simdf_0123to2222( c, cs ); \
6521 stbir__simdf_madd_mem( tot0, tot0, c, decode+8 ); \
6522 stbir__simdf_0123to3333( c, cs ); \
6523 stbir__simdf_madd_mem( tot1, tot1, c, decode+12 );
6524
6525#define stbir__4_coeff_continue_from_4( ofs ) \
6526 STBIR_SIMD_NO_UNROLL(decode); \
6527 stbir__simdf_load( cs, hc + (ofs) ); \
6528 stbir__simdf_0123to0000( c, cs ); \
6529 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*4 ); \
6530 stbir__simdf_0123to1111( c, cs ); \
6531 stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*4+4 ); \
6532 stbir__simdf_0123to2222( c, cs ); \
6533 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*4+8 ); \
6534 stbir__simdf_0123to3333( c, cs ); \
6535 stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*4+12 );
6536
6537#define stbir__1_coeff_remnant( ofs ) \
6538 STBIR_SIMD_NO_UNROLL(decode); \
6539 stbir__simdf_load1( c, hc + (ofs) ); \
6540 stbir__simdf_0123to0000( c, c ); \
6541 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*4 );
6542
6543#define stbir__2_coeff_remnant( ofs ) \
6544 STBIR_SIMD_NO_UNROLL(decode); \
6545 stbir__simdf_load2( cs, hc + (ofs) ); \
6546 stbir__simdf_0123to0000( c, cs ); \
6547 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*4 ); \
6548 stbir__simdf_0123to1111( c, cs ); \
6549 stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*4+4 );
6550
6551#define stbir__3_coeff_remnant( ofs ) \
6552 STBIR_SIMD_NO_UNROLL(decode); \
6553 stbir__simdf_load( cs, hc + (ofs) ); \
6554 stbir__simdf_0123to0000( c, cs ); \
6555 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*4 ); \
6556 stbir__simdf_0123to1111( c, cs ); \
6557 stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*4+4 ); \
6558 stbir__simdf_0123to2222( c, cs ); \
6559 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*4+8 );
6560
6561// polyphase period 1 for 4 channels (when have exact downsamples 2:1, 3:1, etc - common for mipmaps)
6562#define stbir__period1_setup_4() stbir__simdf P0, P1, P2, P3; stbir__simdf_load1frep4( P0, hc[0] ); stbir__simdf_load1frep4( P1, hc[1] ); stbir__simdf_load1frep4( P2, hc[2] ); stbir__simdf_load1frep4( P3, hc[3] );
6563#define stbir__period1_setup_5() stbir__simdf PR; stbir__period1_setup_4() stbir__simdf_load1frep4( PR, hc[4] );
6564#define stbir__period1_setup_8() stbir__simdf P4, P5, P6, P7; stbir__period1_setup_4() stbir__simdf_load1frep4( P4, hc[4] ); stbir__simdf_load1frep4( P5, hc[5] ); stbir__simdf_load1frep4( P6, hc[6] ); stbir__simdf_load1frep4( P7, hc[7] );
6565#define stbir__period1_setup_9() stbir__simdf PR; stbir__period1_setup_8() stbir__simdf_load1frep4( PR, hc[8] );
6566#define stbir__period1_setup_11() stbir__simdf P8, P9, P10; stbir__period1_setup_8() stbir__simdf_load1frep4( P8, hc[8] ); stbir__simdf_load1frep4( P9, hc[9] ); stbir__simdf_load1frep4( P10, hc[10] );
6567#define stbir__period1_setup_12() stbir__simdf P8, P9, P10, P11; stbir__period1_setup_8() stbir__simdf_load1frep4( P8, hc[8] ); stbir__simdf_load1frep4( P9, hc[9] ); stbir__simdf_load1frep4( P10, hc[10] ); stbir__simdf_load1frep4( P11, hc[11] );
6568#define stbir__period1_setup_13() stbir__simdf PR; stbir__period1_setup_12() stbir__simdf_load1frep4( PR, hc[12] );
6569#define stbir__period1_setup_16() stbir__simdf P12, P13, P14, P15; stbir__period1_setup_12() stbir__simdf_load1frep4( P12, hc[12] ); stbir__simdf_load1frep4( P13, hc[13] ); stbir__simdf_load1frep4( P14, hc[14] ); stbir__simdf_load1frep4( P15, hc[15] );
6570#define stbir__period1_setup_17() stbir__simdf PR; stbir__period1_setup_16() stbir__simdf_load1frep4( PR, hc[16] );
6571
6572#define stbir__period1_4_coeff_start() \
6573 stbir__simdf tot0, tot1; \
6574 STBIR_SIMD_NO_UNROLL(decode); \
6575 stbir__simdf_mult_mem( tot0, P0, decode ); \
6576 stbir__simdf_mult_mem( tot1, P1, decode + 4 ); \
6577 stbir__simdf_madd_mem( tot0, tot0, P2, decode + 8 ); \
6578 stbir__simdf_madd_mem( tot1, tot1, P3, decode + 12 );
6579
6580#define stbir__period1_4_coeff_continue() \
6581 stbir__simdf_madd_mem( tot0, tot0, P4, decode + 16 ); \
6582 stbir__simdf_madd_mem( tot1, tot1, P5, decode + 20 ); \
6583 stbir__simdf_madd_mem( tot0, tot0, P6, decode + 24 ); \
6584 stbir__simdf_madd_mem( tot1, tot1, P7, decode + 28 );
6585#define stbir__period1_4_coeff_continue2() \
6586 stbir__simdf_madd_mem( tot0, tot0, P8, decode + 32 ); \
6587 stbir__simdf_madd_mem( tot1, tot1, P9, decode + 36 ); \
6588 stbir__simdf_madd_mem( tot0, tot0, P10, decode + 40 ); \
6589 stbir__simdf_madd_mem( tot1, tot1, P11, decode + 44 );
6590#define stbir__period1_4_coeff_continue3() \
6591 stbir__simdf_madd_mem( tot0, tot0, P12, decode + 48 ); \
6592 stbir__simdf_madd_mem( tot1, tot1, P13, decode + 52 ); \
6593 stbir__simdf_madd_mem( tot0, tot0, P14, decode + 56 ); \
6594 stbir__simdf_madd_mem( tot1, tot1, P15, decode + 60 );
6595
6596#define stbir__period1_1_coeff_remnant( ofs ) \
6597 stbir__simdf_madd_mem( tot0, tot0, PR, decode + (ofs) * 4 );
6598#define stbir__period1_3_coeff_remnant( ofs ) \
6599 stbir__simdf_madd_mem( tot0, tot0, P8, decode + (ofs) * 4 ); \
6600 stbir__simdf_madd_mem( tot1, tot1, P9, decode + (ofs) * 4 + 4 ); \
6601 stbir__simdf_madd_mem( tot0, tot0, P10, decode + (ofs) * 4 + 8 );
6602
6603#define stbir__period1_store_output() \
6604 stbir__simdf_add( tot0, tot0, tot1 ); \
6605 stbir__simdf_store( output, tot0 ); \
6606 decode_buffer += advance; \
6607 output += 4;
6608
6609#define stbir__store_output() \
6610 stbir__simdf_add( tot0, tot0, tot1 ); \
6611 stbir__simdf_store( output, tot0 ); \
6612 stbir__next_contributor(); \
6613 output += 4;
6614
6615#endif
6616
6617#else
6618
6619#define stbir__1_coeff_only() \
6620 float p0,p1,p2,p3,c; \
6621 STBIR_SIMD_NO_UNROLL(decode); \
6622 c = hc[0]; \
6623 p0 = decode[0] * c; \
6624 p1 = decode[1] * c; \
6625 p2 = decode[2] * c; \
6626 p3 = decode[3] * c;
6627
6628#define stbir__2_coeff_only() \
6629 float p0,p1,p2,p3,c; \
6630 STBIR_SIMD_NO_UNROLL(decode); \
6631 c = hc[0]; \
6632 p0 = decode[0] * c; \
6633 p1 = decode[1] * c; \
6634 p2 = decode[2] * c; \
6635 p3 = decode[3] * c; \
6636 c = hc[1]; \
6637 p0 += decode[4] * c; \
6638 p1 += decode[5] * c; \
6639 p2 += decode[6] * c; \
6640 p3 += decode[7] * c;
6641
6642#define stbir__3_coeff_only() \
6643 float p0,p1,p2,p3,c; \
6644 STBIR_SIMD_NO_UNROLL(decode); \
6645 c = hc[0]; \
6646 p0 = decode[0] * c; \
6647 p1 = decode[1] * c; \
6648 p2 = decode[2] * c; \
6649 p3 = decode[3] * c; \
6650 c = hc[1]; \
6651 p0 += decode[4] * c; \
6652 p1 += decode[5] * c; \
6653 p2 += decode[6] * c; \
6654 p3 += decode[7] * c; \
6655 c = hc[2]; \
6656 p0 += decode[8] * c; \
6657 p1 += decode[9] * c; \
6658 p2 += decode[10] * c; \
6659 p3 += decode[11] * c;
6660
6661#define stbir__store_output_tiny() \
6662 output[0] = p0; \
6663 output[1] = p1; \
6664 output[2] = p2; \
6665 output[3] = p3; \
6666 stbir__next_contributor(); \
6667 output += 4;
6668
6669#define stbir__4_coeff_start() \
6670 float x0,x1,x2,x3,y0,y1,y2,y3,c; \
6671 STBIR_SIMD_NO_UNROLL(decode); \
6672 c = hc[0]; \
6673 x0 = decode[0] * c; \
6674 x1 = decode[1] * c; \
6675 x2 = decode[2] * c; \
6676 x3 = decode[3] * c; \
6677 c = hc[1]; \
6678 y0 = decode[4] * c; \
6679 y1 = decode[5] * c; \
6680 y2 = decode[6] * c; \
6681 y3 = decode[7] * c; \
6682 c = hc[2]; \
6683 x0 += decode[8] * c; \
6684 x1 += decode[9] * c; \
6685 x2 += decode[10] * c; \
6686 x3 += decode[11] * c; \
6687 c = hc[3]; \
6688 y0 += decode[12] * c; \
6689 y1 += decode[13] * c; \
6690 y2 += decode[14] * c; \
6691 y3 += decode[15] * c;
6692
6693#define stbir__4_coeff_continue_from_4( ofs ) \
6694 STBIR_SIMD_NO_UNROLL(decode); \
6695 c = hc[0+(ofs)]; \
6696 x0 += decode[0+(ofs)*4] * c; \
6697 x1 += decode[1+(ofs)*4] * c; \
6698 x2 += decode[2+(ofs)*4] * c; \
6699 x3 += decode[3+(ofs)*4] * c; \
6700 c = hc[1+(ofs)]; \
6701 y0 += decode[4+(ofs)*4] * c; \
6702 y1 += decode[5+(ofs)*4] * c; \
6703 y2 += decode[6+(ofs)*4] * c; \
6704 y3 += decode[7+(ofs)*4] * c; \
6705 c = hc[2+(ofs)]; \
6706 x0 += decode[8+(ofs)*4] * c; \
6707 x1 += decode[9+(ofs)*4] * c; \
6708 x2 += decode[10+(ofs)*4] * c; \
6709 x3 += decode[11+(ofs)*4] * c; \
6710 c = hc[3+(ofs)]; \
6711 y0 += decode[12+(ofs)*4] * c; \
6712 y1 += decode[13+(ofs)*4] * c; \
6713 y2 += decode[14+(ofs)*4] * c; \
6714 y3 += decode[15+(ofs)*4] * c;
6715
6716#define stbir__1_coeff_remnant( ofs ) \
6717 STBIR_SIMD_NO_UNROLL(decode); \
6718 c = hc[0+(ofs)]; \
6719 x0 += decode[0+(ofs)*4] * c; \
6720 x1 += decode[1+(ofs)*4] * c; \
6721 x2 += decode[2+(ofs)*4] * c; \
6722 x3 += decode[3+(ofs)*4] * c;
6723
6724#define stbir__2_coeff_remnant( ofs ) \
6725 STBIR_SIMD_NO_UNROLL(decode); \
6726 c = hc[0+(ofs)]; \
6727 x0 += decode[0+(ofs)*4] * c; \
6728 x1 += decode[1+(ofs)*4] * c; \
6729 x2 += decode[2+(ofs)*4] * c; \
6730 x3 += decode[3+(ofs)*4] * c; \
6731 c = hc[1+(ofs)]; \
6732 y0 += decode[4+(ofs)*4] * c; \
6733 y1 += decode[5+(ofs)*4] * c; \
6734 y2 += decode[6+(ofs)*4] * c; \
6735 y3 += decode[7+(ofs)*4] * c;
6736
6737#define stbir__3_coeff_remnant( ofs ) \
6738 STBIR_SIMD_NO_UNROLL(decode); \
6739 c = hc[0+(ofs)]; \
6740 x0 += decode[0+(ofs)*4] * c; \
6741 x1 += decode[1+(ofs)*4] * c; \
6742 x2 += decode[2+(ofs)*4] * c; \
6743 x3 += decode[3+(ofs)*4] * c; \
6744 c = hc[1+(ofs)]; \
6745 y0 += decode[4+(ofs)*4] * c; \
6746 y1 += decode[5+(ofs)*4] * c; \
6747 y2 += decode[6+(ofs)*4] * c; \
6748 y3 += decode[7+(ofs)*4] * c; \
6749 c = hc[2+(ofs)]; \
6750 x0 += decode[8+(ofs)*4] * c; \
6751 x1 += decode[9+(ofs)*4] * c; \
6752 x2 += decode[10+(ofs)*4] * c; \
6753 x3 += decode[11+(ofs)*4] * c;
6754
6755#define stbir__store_output() \
6756 output[0] = x0 + y0; \
6757 output[1] = x1 + y1; \
6758 output[2] = x2 + y2; \
6759 output[3] = x3 + y3; \
6760 stbir__next_contributor(); \
6761 output += 4;
6762
6763#endif
6764
6765#define STBIR__horizontal_channels 4
6766#define STB_IMAGE_RESIZE_DO_HORIZONTALS
6767#include STBIR__HEADER_FILENAME
6768
6769
6770//=================
6771// Do 7 channel horizontal routines
6772
6773#ifdef STBIR_SIMD
6774
6775#define stbir__1_coeff_only() \
6776 stbir__simdf tot0,tot1,c; \
6777 STBIR_SIMD_NO_UNROLL(decode); \
6778 stbir__simdf_load1( c, hc ); \
6779 stbir__simdf_0123to0000( c, c ); \
6780 stbir__simdf_mult_mem( tot0, c, decode ); \
6781 stbir__simdf_mult_mem( tot1, c, decode+3 );
6782
6783#define stbir__2_coeff_only() \
6784 stbir__simdf tot0,tot1,c,cs; \
6785 STBIR_SIMD_NO_UNROLL(decode); \
6786 stbir__simdf_load2( cs, hc ); \
6787 stbir__simdf_0123to0000( c, cs ); \
6788 stbir__simdf_mult_mem( tot0, c, decode ); \
6789 stbir__simdf_mult_mem( tot1, c, decode+3 ); \
6790 stbir__simdf_0123to1111( c, cs ); \
6791 stbir__simdf_madd_mem( tot0, tot0, c, decode+7 ); \
6792 stbir__simdf_madd_mem( tot1, tot1, c,decode+10 );
6793
6794#define stbir__3_coeff_only() \
6795 stbir__simdf tot0,tot1,c,cs; \
6796 STBIR_SIMD_NO_UNROLL(decode); \
6797 stbir__simdf_load( cs, hc ); \
6798 stbir__simdf_0123to0000( c, cs ); \
6799 stbir__simdf_mult_mem( tot0, c, decode ); \
6800 stbir__simdf_mult_mem( tot1, c, decode+3 ); \
6801 stbir__simdf_0123to1111( c, cs ); \
6802 stbir__simdf_madd_mem( tot0, tot0, c, decode+7 ); \
6803 stbir__simdf_madd_mem( tot1, tot1, c, decode+10 ); \
6804 stbir__simdf_0123to2222( c, cs ); \
6805 stbir__simdf_madd_mem( tot0, tot0, c, decode+14 ); \
6806 stbir__simdf_madd_mem( tot1, tot1, c, decode+17 );
6807
6808#define stbir__store_output_tiny() \
6809 stbir__simdf_store( output+3, tot1 ); \
6810 stbir__simdf_store( output, tot0 ); \
6811 stbir__next_contributor(); \
6812 output += 7;
6813
6814#ifdef STBIR_SIMD8
6815
6816#define stbir__4_coeff_start() \
6817 stbir__simdf8 tot0,tot1,c,cs; \
6818 STBIR_SIMD_NO_UNROLL(decode); \
6819 stbir__simdf8_load4b( cs, hc ); \
6820 stbir__simdf8_0123to00000000( c, cs ); \
6821 stbir__simdf8_mult_mem( tot0, c, decode ); \
6822 stbir__simdf8_0123to11111111( c, cs ); \
6823 stbir__simdf8_mult_mem( tot1, c, decode+7 ); \
6824 stbir__simdf8_0123to22222222( c, cs ); \
6825 stbir__simdf8_madd_mem( tot0, tot0, c, decode+14 ); \
6826 stbir__simdf8_0123to33333333( c, cs ); \
6827 stbir__simdf8_madd_mem( tot1, tot1, c, decode+21 );
6828
6829#define stbir__4_coeff_continue_from_4( ofs ) \
6830 STBIR_SIMD_NO_UNROLL(decode); \
6831 stbir__simdf8_load4b( cs, hc + (ofs) ); \
6832 stbir__simdf8_0123to00000000( c, cs ); \
6833 stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*7 ); \
6834 stbir__simdf8_0123to11111111( c, cs ); \
6835 stbir__simdf8_madd_mem( tot1, tot1, c, decode+(ofs)*7+7 ); \
6836 stbir__simdf8_0123to22222222( c, cs ); \
6837 stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*7+14 ); \
6838 stbir__simdf8_0123to33333333( c, cs ); \
6839 stbir__simdf8_madd_mem( tot1, tot1, c, decode+(ofs)*7+21 );
6840
6841#define stbir__1_coeff_remnant( ofs ) \
6842 STBIR_SIMD_NO_UNROLL(decode); \
6843 stbir__simdf8_load1b( c, hc + (ofs) ); \
6844 stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*7 );
6845
6846#define stbir__2_coeff_remnant( ofs ) \
6847 STBIR_SIMD_NO_UNROLL(decode); \
6848 stbir__simdf8_load1b( c, hc + (ofs) ); \
6849 stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*7 ); \
6850 stbir__simdf8_load1b( c, hc + (ofs)+1 ); \
6851 stbir__simdf8_madd_mem( tot1, tot1, c, decode+(ofs)*7+7 );
6852
6853#define stbir__3_coeff_remnant( ofs ) \
6854 STBIR_SIMD_NO_UNROLL(decode); \
6855 stbir__simdf8_load4b( cs, hc + (ofs) ); \
6856 stbir__simdf8_0123to00000000( c, cs ); \
6857 stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*7 ); \
6858 stbir__simdf8_0123to11111111( c, cs ); \
6859 stbir__simdf8_madd_mem( tot1, tot1, c, decode+(ofs)*7+7 ); \
6860 stbir__simdf8_0123to22222222( c, cs ); \
6861 stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*7+14 );
6862
6863// polyphase period 1 for 7 channels (when have exact downsamples 2:1, 3:1, etc - common for mipmaps)
6864#define stbir__period1_setup_4() stbir__simdf8 P0, P1, P2, P3; stbir__simdf8_load1b( P0, hc ); stbir__simdf8_load1b( P1, hc + 1 ); stbir__simdf8_load1b( P2, hc + 2 ); stbir__simdf8_load1b( P3, hc + 3 );
6865#define stbir__period1_setup_5() stbir__simdf8 PR; stbir__period1_setup_4() stbir__simdf8_load1b( PR, hc + 4 );
6866#define stbir__period1_setup_8() stbir__simdf8 P4, P5, P6, P7; stbir__period1_setup_4() stbir__simdf8_load1b( P4, hc + 4 ); stbir__simdf8_load1b( P5, hc + 5 ); stbir__simdf8_load1b( P6, hc + 6 ); stbir__simdf8_load1b( P7, hc + 7 );
6867#define stbir__period1_setup_9() stbir__simdf8 PR; stbir__period1_setup_8() stbir__simdf8_load1b( PR, hc + 8 );
6868#define stbir__period1_setup_11() stbir__simdf8 P8, P9, P10; stbir__period1_setup_8() stbir__simdf8_load1b( P8, hc + 8 ); stbir__simdf8_load1b( P9, hc + 9 ); stbir__simdf8_load1b( P10, hc + 10 );
6869#define stbir__period1_setup_12() stbir__simdf8 P8, P9, P10, P11; stbir__period1_setup_8() stbir__simdf8_load1b( P8, hc + 8 ); stbir__simdf8_load1b( P9, hc + 9 ); stbir__simdf8_load1b( P10, hc + 10 ); stbir__simdf8_load1b( P11, hc + 11 );
6870#define stbir__period1_setup_13() stbir__simdf8 PR; stbir__period1_setup_12() stbir__simdf8_load1b( PR, hc + 12 );
6871#define stbir__period1_setup_16() stbir__simdf8 P12, P13, P14, P15; stbir__period1_setup_12() stbir__simdf8_load1b( P12, hc + 12 ); stbir__simdf8_load1b( P13, hc + 13 ); stbir__simdf8_load1b( P14, hc + 14 ); stbir__simdf8_load1b( P15, hc + 15 );
6872#define stbir__period1_setup_17() stbir__simdf8 PR; stbir__period1_setup_16() stbir__simdf8_load1b( PR, hc + 16 );
6873
6874#define stbir__period1_4_coeff_start() \
6875 stbir__simdf8 tot0, tot1; \
6876 STBIR_SIMD_NO_UNROLL(decode); \
6877 stbir__simdf8_mult_mem( tot0, P0, decode ); \
6878 stbir__simdf8_mult_mem( tot1, P1, decode + 7 ); \
6879 stbir__simdf8_madd_mem( tot0, tot0, P2, decode + 14 ); \
6880 stbir__simdf8_madd_mem( tot1, tot1, P3, decode + 21 );
6881
6882#define stbir__period1_4_coeff_continue() \
6883 stbir__simdf8_madd_mem( tot0, tot0, P4, decode + 28 ); \
6884 stbir__simdf8_madd_mem( tot1, tot1, P5, decode + 35 ); \
6885 stbir__simdf8_madd_mem( tot0, tot0, P6, decode + 42 ); \
6886 stbir__simdf8_madd_mem( tot1, tot1, P7, decode + 49 );
6887#define stbir__period1_4_coeff_continue2() \
6888 stbir__simdf8_madd_mem( tot0, tot0, P8, decode + 56 ); \
6889 stbir__simdf8_madd_mem( tot1, tot1, P9, decode + 63 ); \
6890 stbir__simdf8_madd_mem( tot0, tot0, P10, decode + 70 ); \
6891 stbir__simdf8_madd_mem( tot1, tot1, P11, decode + 77 );
6892#define stbir__period1_4_coeff_continue3() \
6893 stbir__simdf8_madd_mem( tot0, tot0, P12, decode + 84 ); \
6894 stbir__simdf8_madd_mem( tot1, tot1, P13, decode + 91 ); \
6895 stbir__simdf8_madd_mem( tot0, tot0, P14, decode + 98 ); \
6896 stbir__simdf8_madd_mem( tot1, tot1, P15, decode + 105 );
6897
6898#define stbir__period1_1_coeff_remnant( ofs ) \
6899 stbir__simdf8_madd_mem( tot0, tot0, PR, decode + (ofs) * 7 );
6900#define stbir__period1_3_coeff_remnant( ofs ) \
6901 stbir__simdf8_madd_mem( tot0, tot0, P8, decode + (ofs) * 7 ); \
6902 stbir__simdf8_madd_mem( tot1, tot1, P9, decode + (ofs) * 7 + 7 ); \
6903 stbir__simdf8_madd_mem( tot0, tot0, P10, decode + (ofs) * 7 + 14 );
6904
6905#define stbir__period1_store_output() \
6906 stbir__simdf8_add( tot0, tot0, tot1 ); \
6907 decode_buffer += advance; \
6908 output += 7; \
6909 if ( output < output_end ) \
6910 { \
6911 stbir__simdf8_store( output-7, tot0 ); \
6912 continue; \
6913 } \
6914 stbir__simdf_store( output-7+3, stbir__simdf_swiz(stbir__simdf8_gettop4(tot0),0,0,1,2) ); \
6915 stbir__simdf_store( output-7, stbir__if_simdf8_cast_to_simdf4(tot0) ); \
6916 break;
6917
6918#define stbir__store_output() \
6919 stbir__simdf8_add( tot0, tot0, tot1 ); \
6920 stbir__next_contributor(); \
6921 output += 7; \
6922 if ( output < output_end ) \
6923 { \
6924 stbir__simdf8_store( output-7, tot0 ); \
6925 continue; \
6926 } \
6927 stbir__simdf_store( output-7+3, stbir__simdf_swiz(stbir__simdf8_gettop4(tot0),0,0,1,2) ); \
6928 stbir__simdf_store( output-7, stbir__if_simdf8_cast_to_simdf4(tot0) ); \
6929 break;
6930
6931#else
6932
6933#define stbir__4_coeff_start() \
6934 stbir__simdf tot0,tot1,tot2,tot3,c,cs; \
6935 STBIR_SIMD_NO_UNROLL(decode); \
6936 stbir__simdf_load( cs, hc ); \
6937 stbir__simdf_0123to0000( c, cs ); \
6938 stbir__simdf_mult_mem( tot0, c, decode ); \
6939 stbir__simdf_mult_mem( tot1, c, decode+3 ); \
6940 stbir__simdf_0123to1111( c, cs ); \
6941 stbir__simdf_mult_mem( tot2, c, decode+7 ); \
6942 stbir__simdf_mult_mem( tot3, c, decode+10 ); \
6943 stbir__simdf_0123to2222( c, cs ); \
6944 stbir__simdf_madd_mem( tot0, tot0, c, decode+14 ); \
6945 stbir__simdf_madd_mem( tot1, tot1, c, decode+17 ); \
6946 stbir__simdf_0123to3333( c, cs ); \
6947 stbir__simdf_madd_mem( tot2, tot2, c, decode+21 ); \
6948 stbir__simdf_madd_mem( tot3, tot3, c, decode+24 );
6949
6950#define stbir__4_coeff_continue_from_4( ofs ) \
6951 STBIR_SIMD_NO_UNROLL(decode); \
6952 stbir__simdf_load( cs, hc + (ofs) ); \
6953 stbir__simdf_0123to0000( c, cs ); \
6954 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*7 ); \
6955 stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*7+3 ); \
6956 stbir__simdf_0123to1111( c, cs ); \
6957 stbir__simdf_madd_mem( tot2, tot2, c, decode+(ofs)*7+7 ); \
6958 stbir__simdf_madd_mem( tot3, tot3, c, decode+(ofs)*7+10 ); \
6959 stbir__simdf_0123to2222( c, cs ); \
6960 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*7+14 ); \
6961 stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*7+17 ); \
6962 stbir__simdf_0123to3333( c, cs ); \
6963 stbir__simdf_madd_mem( tot2, tot2, c, decode+(ofs)*7+21 ); \
6964 stbir__simdf_madd_mem( tot3, tot3, c, decode+(ofs)*7+24 );
6965
6966#define stbir__1_coeff_remnant( ofs ) \
6967 STBIR_SIMD_NO_UNROLL(decode); \
6968 stbir__simdf_load1( c, hc + (ofs) ); \
6969 stbir__simdf_0123to0000( c, c ); \
6970 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*7 ); \
6971 stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*7+3 ); \
6972
6973#define stbir__2_coeff_remnant( ofs ) \
6974 STBIR_SIMD_NO_UNROLL(decode); \
6975 stbir__simdf_load2( cs, hc + (ofs) ); \
6976 stbir__simdf_0123to0000( c, cs ); \
6977 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*7 ); \
6978 stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*7+3 ); \
6979 stbir__simdf_0123to1111( c, cs ); \
6980 stbir__simdf_madd_mem( tot2, tot2, c, decode+(ofs)*7+7 ); \
6981 stbir__simdf_madd_mem( tot3, tot3, c, decode+(ofs)*7+10 );
6982
6983#define stbir__3_coeff_remnant( ofs ) \
6984 STBIR_SIMD_NO_UNROLL(decode); \
6985 stbir__simdf_load( cs, hc + (ofs) ); \
6986 stbir__simdf_0123to0000( c, cs ); \
6987 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*7 ); \
6988 stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*7+3 ); \
6989 stbir__simdf_0123to1111( c, cs ); \
6990 stbir__simdf_madd_mem( tot2, tot2, c, decode+(ofs)*7+7 ); \
6991 stbir__simdf_madd_mem( tot3, tot3, c, decode+(ofs)*7+10 ); \
6992 stbir__simdf_0123to2222( c, cs ); \
6993 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*7+14 ); \
6994 stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*7+17 );
6995
6996// polyphase period 1 for 7 channels (when have exact downsamples 2:1, 3:1, etc - common for mipmaps)
6997#define stbir__period1_setup_4() stbir__simdf P0, P1, P2, P3; stbir__simdf_load1frep4( P0, hc[0] ); stbir__simdf_load1frep4( P1, hc[1] ); stbir__simdf_load1frep4( P2, hc[2] ); stbir__simdf_load1frep4( P3, hc[3] );
6998#define stbir__period1_setup_5() stbir__simdf PR; stbir__period1_setup_4() stbir__simdf_load1frep4( PR, hc[4] );
6999#define stbir__period1_setup_8() stbir__simdf P4, P5, P6, P7; stbir__period1_setup_4() stbir__simdf_load1frep4( P4, hc[4] ); stbir__simdf_load1frep4( P5, hc[5] ); stbir__simdf_load1frep4( P6, hc[6] ); stbir__simdf_load1frep4( P7, hc[7] );
7000#define stbir__period1_setup_9() stbir__simdf PR; stbir__period1_setup_8() stbir__simdf_load1frep4( PR, hc[8] );
7001#define stbir__period1_setup_11() stbir__simdf P8, P9, P10; stbir__period1_setup_8() stbir__simdf_load1frep4( P8, hc[8] ); stbir__simdf_load1frep4( P9, hc[9] ); stbir__simdf_load1frep4( P10, hc[10] );
7002#define stbir__period1_setup_12() stbir__simdf P8, P9, P10, P11; stbir__period1_setup_8() stbir__simdf_load1frep4( P8, hc[8] ); stbir__simdf_load1frep4( P9, hc[9] ); stbir__simdf_load1frep4( P10, hc[10] ); stbir__simdf_load1frep4( P11, hc[11] );
7003#define stbir__period1_setup_13() stbir__simdf PR; stbir__period1_setup_12() stbir__simdf_load1frep4( PR, hc[12] );
7004#define stbir__period1_setup_16() stbir__simdf P12, P13, P14, P15; stbir__period1_setup_12() stbir__simdf_load1frep4( P12, hc[12] ); stbir__simdf_load1frep4( P13, hc[13] ); stbir__simdf_load1frep4( P14, hc[14] ); stbir__simdf_load1frep4( P15, hc[15] );
7005#define stbir__period1_setup_17() stbir__simdf PR; stbir__period1_setup_16() stbir__simdf_load1frep4( PR, hc[16] );
7006
7007#define stbir__period1_4_coeff_start() \
7008 stbir__simdf tot0, tot1, tot2, tot3; \
7009 STBIR_SIMD_NO_UNROLL(decode); \
7010 stbir__simdf_mult_mem( tot0, P0, decode ); \
7011 stbir__simdf_mult_mem( tot1, P0, decode + 3 ); \
7012 stbir__simdf_mult_mem( tot2, P1, decode + 7 ); \
7013 stbir__simdf_mult_mem( tot3, P1, decode + 10 ); \
7014 stbir__simdf_madd_mem( tot0, tot0, P2, decode + 14 ); \
7015 stbir__simdf_madd_mem( tot1, tot1, P2, decode + 17 ); \
7016 stbir__simdf_madd_mem( tot2, tot2, P3, decode + 21 ); \
7017 stbir__simdf_madd_mem( tot3, tot3, P3, decode + 24 );
7018
7019#define stbir__period1_4_coeff_continue() \
7020 stbir__simdf_madd_mem( tot0, tot0, P4, decode + 28 ); \
7021 stbir__simdf_madd_mem( tot1, tot1, P4, decode + 31 ); \
7022 stbir__simdf_madd_mem( tot2, tot2, P5, decode + 35 ); \
7023 stbir__simdf_madd_mem( tot3, tot3, P5, decode + 38 ); \
7024 stbir__simdf_madd_mem( tot0, tot0, P6, decode + 42 ); \
7025 stbir__simdf_madd_mem( tot1, tot1, P6, decode + 45 ); \
7026 stbir__simdf_madd_mem( tot2, tot2, P7, decode + 49 ); \
7027 stbir__simdf_madd_mem( tot3, tot3, P7, decode + 52 );
7028#define stbir__period1_4_coeff_continue2() \
7029 stbir__simdf_madd_mem( tot0, tot0, P8, decode + 56 ); \
7030 stbir__simdf_madd_mem( tot1, tot1, P8, decode + 59 ); \
7031 stbir__simdf_madd_mem( tot2, tot2, P9, decode + 63 ); \
7032 stbir__simdf_madd_mem( tot3, tot3, P9, decode + 66 ); \
7033 stbir__simdf_madd_mem( tot0, tot0, P10, decode + 70 ); \
7034 stbir__simdf_madd_mem( tot1, tot1, P10, decode + 73 ); \
7035 stbir__simdf_madd_mem( tot2, tot2, P11, decode + 77 ); \
7036 stbir__simdf_madd_mem( tot3, tot3, P11, decode + 80 );
7037#define stbir__period1_4_coeff_continue3() \
7038 stbir__simdf_madd_mem( tot0, tot0, P12, decode + 84 ); \
7039 stbir__simdf_madd_mem( tot1, tot1, P12, decode + 87 ); \
7040 stbir__simdf_madd_mem( tot2, tot2, P13, decode + 91 ); \
7041 stbir__simdf_madd_mem( tot3, tot3, P13, decode + 94 ); \
7042 stbir__simdf_madd_mem( tot0, tot0, P14, decode + 98 ); \
7043 stbir__simdf_madd_mem( tot1, tot1, P14, decode + 101 ); \
7044 stbir__simdf_madd_mem( tot2, tot2, P15, decode + 105 ); \
7045 stbir__simdf_madd_mem( tot3, tot3, P15, decode + 108 );
7046
7047#define stbir__period1_1_coeff_remnant( ofs ) \
7048 stbir__simdf_madd_mem( tot0, tot0, PR, decode + (ofs) * 7 ); \
7049 stbir__simdf_madd_mem( tot1, tot1, PR, decode + (ofs) * 7 + 3 );
7050#define stbir__period1_3_coeff_remnant( ofs ) \
7051 stbir__simdf_madd_mem( tot0, tot0, P8, decode + (ofs) * 7 ); \
7052 stbir__simdf_madd_mem( tot1, tot1, P8, decode + (ofs) * 7 + 3 ); \
7053 stbir__simdf_madd_mem( tot2, tot2, P9, decode + (ofs) * 7 + 7 ); \
7054 stbir__simdf_madd_mem( tot3, tot3, P9, decode + (ofs) * 7 + 10 ); \
7055 stbir__simdf_madd_mem( tot0, tot0, P10, decode + (ofs) * 7 + 14 ); \
7056 stbir__simdf_madd_mem( tot1, tot1, P10, decode + (ofs) * 7 + 17 );
7057
7058#define stbir__period1_store_output() \
7059 stbir__simdf_add( tot0, tot0, tot2 ); \
7060 stbir__simdf_add( tot1, tot1, tot3 ); \
7061 stbir__simdf_store( output+3, tot1 ); \
7062 stbir__simdf_store( output, tot0 ); \
7063 decode_buffer += advance; \
7064 output += 7;
7065
7066#define stbir__store_output() \
7067 stbir__simdf_add( tot0, tot0, tot2 ); \
7068 stbir__simdf_add( tot1, tot1, tot3 ); \
7069 stbir__simdf_store( output+3, tot1 ); \
7070 stbir__simdf_store( output, tot0 ); \
7071 stbir__next_contributor(); \
7072 output += 7;
7073
7074#endif
7075
7076#else
7077
7078#define stbir__1_coeff_only() \
7079 float tot0, tot1, tot2, tot3, tot4, tot5, tot6, c; \
7080 c = hc[0]; \
7081 tot0 = decode[0]*c; \
7082 tot1 = decode[1]*c; \
7083 tot2 = decode[2]*c; \
7084 tot3 = decode[3]*c; \
7085 tot4 = decode[4]*c; \
7086 tot5 = decode[5]*c; \
7087 tot6 = decode[6]*c;
7088
7089#define stbir__2_coeff_only() \
7090 float tot0, tot1, tot2, tot3, tot4, tot5, tot6, c; \
7091 c = hc[0]; \
7092 tot0 = decode[0]*c; \
7093 tot1 = decode[1]*c; \
7094 tot2 = decode[2]*c; \
7095 tot3 = decode[3]*c; \
7096 tot4 = decode[4]*c; \
7097 tot5 = decode[5]*c; \
7098 tot6 = decode[6]*c; \
7099 c = hc[1]; \
7100 tot0 += decode[7]*c; \
7101 tot1 += decode[8]*c; \
7102 tot2 += decode[9]*c; \
7103 tot3 += decode[10]*c; \
7104 tot4 += decode[11]*c; \
7105 tot5 += decode[12]*c; \
7106 tot6 += decode[13]*c; \
7107
7108#define stbir__3_coeff_only() \
7109 float tot0, tot1, tot2, tot3, tot4, tot5, tot6, c; \
7110 c = hc[0]; \
7111 tot0 = decode[0]*c; \
7112 tot1 = decode[1]*c; \
7113 tot2 = decode[2]*c; \
7114 tot3 = decode[3]*c; \
7115 tot4 = decode[4]*c; \
7116 tot5 = decode[5]*c; \
7117 tot6 = decode[6]*c; \
7118 c = hc[1]; \
7119 tot0 += decode[7]*c; \
7120 tot1 += decode[8]*c; \
7121 tot2 += decode[9]*c; \
7122 tot3 += decode[10]*c; \
7123 tot4 += decode[11]*c; \
7124 tot5 += decode[12]*c; \
7125 tot6 += decode[13]*c; \
7126 c = hc[2]; \
7127 tot0 += decode[14]*c; \
7128 tot1 += decode[15]*c; \
7129 tot2 += decode[16]*c; \
7130 tot3 += decode[17]*c; \
7131 tot4 += decode[18]*c; \
7132 tot5 += decode[19]*c; \
7133 tot6 += decode[20]*c; \
7134
7135#define stbir__store_output_tiny() \
7136 output[0] = tot0; \
7137 output[1] = tot1; \
7138 output[2] = tot2; \
7139 output[3] = tot3; \
7140 output[4] = tot4; \
7141 output[5] = tot5; \
7142 output[6] = tot6; \
7143 stbir__next_contributor(); \
7144 output += 7;
7145
7146#define stbir__4_coeff_start() \
7147 float x0,x1,x2,x3,x4,x5,x6,y0,y1,y2,y3,y4,y5,y6,c; \
7148 STBIR_SIMD_NO_UNROLL(decode); \
7149 c = hc[0]; \
7150 x0 = decode[0] * c; \
7151 x1 = decode[1] * c; \
7152 x2 = decode[2] * c; \
7153 x3 = decode[3] * c; \
7154 x4 = decode[4] * c; \
7155 x5 = decode[5] * c; \
7156 x6 = decode[6] * c; \
7157 c = hc[1]; \
7158 y0 = decode[7] * c; \
7159 y1 = decode[8] * c; \
7160 y2 = decode[9] * c; \
7161 y3 = decode[10] * c; \
7162 y4 = decode[11] * c; \
7163 y5 = decode[12] * c; \
7164 y6 = decode[13] * c; \
7165 c = hc[2]; \
7166 x0 += decode[14] * c; \
7167 x1 += decode[15] * c; \
7168 x2 += decode[16] * c; \
7169 x3 += decode[17] * c; \
7170 x4 += decode[18] * c; \
7171 x5 += decode[19] * c; \
7172 x6 += decode[20] * c; \
7173 c = hc[3]; \
7174 y0 += decode[21] * c; \
7175 y1 += decode[22] * c; \
7176 y2 += decode[23] * c; \
7177 y3 += decode[24] * c; \
7178 y4 += decode[25] * c; \
7179 y5 += decode[26] * c; \
7180 y6 += decode[27] * c;
7181
7182#define stbir__4_coeff_continue_from_4( ofs ) \
7183 STBIR_SIMD_NO_UNROLL(decode); \
7184 c = hc[0+(ofs)]; \
7185 x0 += decode[0+(ofs)*7] * c; \
7186 x1 += decode[1+(ofs)*7] * c; \
7187 x2 += decode[2+(ofs)*7] * c; \
7188 x3 += decode[3+(ofs)*7] * c; \
7189 x4 += decode[4+(ofs)*7] * c; \
7190 x5 += decode[5+(ofs)*7] * c; \
7191 x6 += decode[6+(ofs)*7] * c; \
7192 c = hc[1+(ofs)]; \
7193 y0 += decode[7+(ofs)*7] * c; \
7194 y1 += decode[8+(ofs)*7] * c; \
7195 y2 += decode[9+(ofs)*7] * c; \
7196 y3 += decode[10+(ofs)*7] * c; \
7197 y4 += decode[11+(ofs)*7] * c; \
7198 y5 += decode[12+(ofs)*7] * c; \
7199 y6 += decode[13+(ofs)*7] * c; \
7200 c = hc[2+(ofs)]; \
7201 x0 += decode[14+(ofs)*7] * c; \
7202 x1 += decode[15+(ofs)*7] * c; \
7203 x2 += decode[16+(ofs)*7] * c; \
7204 x3 += decode[17+(ofs)*7] * c; \
7205 x4 += decode[18+(ofs)*7] * c; \
7206 x5 += decode[19+(ofs)*7] * c; \
7207 x6 += decode[20+(ofs)*7] * c; \
7208 c = hc[3+(ofs)]; \
7209 y0 += decode[21+(ofs)*7] * c; \
7210 y1 += decode[22+(ofs)*7] * c; \
7211 y2 += decode[23+(ofs)*7] * c; \
7212 y3 += decode[24+(ofs)*7] * c; \
7213 y4 += decode[25+(ofs)*7] * c; \
7214 y5 += decode[26+(ofs)*7] * c; \
7215 y6 += decode[27+(ofs)*7] * c;
7216
7217#define stbir__1_coeff_remnant( ofs ) \
7218 STBIR_SIMD_NO_UNROLL(decode); \
7219 c = hc[0+(ofs)]; \
7220 x0 += decode[0+(ofs)*7] * c; \
7221 x1 += decode[1+(ofs)*7] * c; \
7222 x2 += decode[2+(ofs)*7] * c; \
7223 x3 += decode[3+(ofs)*7] * c; \
7224 x4 += decode[4+(ofs)*7] * c; \
7225 x5 += decode[5+(ofs)*7] * c; \
7226 x6 += decode[6+(ofs)*7] * c; \
7227
7228#define stbir__2_coeff_remnant( ofs ) \
7229 STBIR_SIMD_NO_UNROLL(decode); \
7230 c = hc[0+(ofs)]; \
7231 x0 += decode[0+(ofs)*7] * c; \
7232 x1 += decode[1+(ofs)*7] * c; \
7233 x2 += decode[2+(ofs)*7] * c; \
7234 x3 += decode[3+(ofs)*7] * c; \
7235 x4 += decode[4+(ofs)*7] * c; \
7236 x5 += decode[5+(ofs)*7] * c; \
7237 x6 += decode[6+(ofs)*7] * c; \
7238 c = hc[1+(ofs)]; \
7239 y0 += decode[7+(ofs)*7] * c; \
7240 y1 += decode[8+(ofs)*7] * c; \
7241 y2 += decode[9+(ofs)*7] * c; \
7242 y3 += decode[10+(ofs)*7] * c; \
7243 y4 += decode[11+(ofs)*7] * c; \
7244 y5 += decode[12+(ofs)*7] * c; \
7245 y6 += decode[13+(ofs)*7] * c; \
7246
7247#define stbir__3_coeff_remnant( ofs ) \
7248 STBIR_SIMD_NO_UNROLL(decode); \
7249 c = hc[0+(ofs)]; \
7250 x0 += decode[0+(ofs)*7] * c; \
7251 x1 += decode[1+(ofs)*7] * c; \
7252 x2 += decode[2+(ofs)*7] * c; \
7253 x3 += decode[3+(ofs)*7] * c; \
7254 x4 += decode[4+(ofs)*7] * c; \
7255 x5 += decode[5+(ofs)*7] * c; \
7256 x6 += decode[6+(ofs)*7] * c; \
7257 c = hc[1+(ofs)]; \
7258 y0 += decode[7+(ofs)*7] * c; \
7259 y1 += decode[8+(ofs)*7] * c; \
7260 y2 += decode[9+(ofs)*7] * c; \
7261 y3 += decode[10+(ofs)*7] * c; \
7262 y4 += decode[11+(ofs)*7] * c; \
7263 y5 += decode[12+(ofs)*7] * c; \
7264 y6 += decode[13+(ofs)*7] * c; \
7265 c = hc[2+(ofs)]; \
7266 x0 += decode[14+(ofs)*7] * c; \
7267 x1 += decode[15+(ofs)*7] * c; \
7268 x2 += decode[16+(ofs)*7] * c; \
7269 x3 += decode[17+(ofs)*7] * c; \
7270 x4 += decode[18+(ofs)*7] * c; \
7271 x5 += decode[19+(ofs)*7] * c; \
7272 x6 += decode[20+(ofs)*7] * c; \
7273
7274#define stbir__store_output() \
7275 output[0] = x0 + y0; \
7276 output[1] = x1 + y1; \
7277 output[2] = x2 + y2; \
7278 output[3] = x3 + y3; \
7279 output[4] = x4 + y4; \
7280 output[5] = x5 + y5; \
7281 output[6] = x6 + y6; \
7282 stbir__next_contributor(); \
7283 output += 7;
7284
7285#endif
7286
7287#define STBIR__horizontal_channels 7
7288#define STB_IMAGE_RESIZE_DO_HORIZONTALS
7289#include STBIR__HEADER_FILENAME
7290
7291
7292// include all of the vertical resamplers (both scatter and gather versions)
7293
7294#define STBIR__vertical_channels 1
7295#define STB_IMAGE_RESIZE_DO_VERTICALS
7296#include STBIR__HEADER_FILENAME
7297
7298#define STBIR__vertical_channels 1
7299#define STB_IMAGE_RESIZE_DO_VERTICALS
7300#define STB_IMAGE_RESIZE_VERTICAL_CONTINUE
7301#include STBIR__HEADER_FILENAME
7302
7303#define STBIR__vertical_channels 2
7304#define STB_IMAGE_RESIZE_DO_VERTICALS
7305#include STBIR__HEADER_FILENAME
7306
7307#define STBIR__vertical_channels 2
7308#define STB_IMAGE_RESIZE_DO_VERTICALS
7309#define STB_IMAGE_RESIZE_VERTICAL_CONTINUE
7310#include STBIR__HEADER_FILENAME
7311
7312#define STBIR__vertical_channels 3
7313#define STB_IMAGE_RESIZE_DO_VERTICALS
7314#include STBIR__HEADER_FILENAME
7315
7316#define STBIR__vertical_channels 3
7317#define STB_IMAGE_RESIZE_DO_VERTICALS
7318#define STB_IMAGE_RESIZE_VERTICAL_CONTINUE
7319#include STBIR__HEADER_FILENAME
7320
7321#define STBIR__vertical_channels 4
7322#define STB_IMAGE_RESIZE_DO_VERTICALS
7323#include STBIR__HEADER_FILENAME
7324
7325#define STBIR__vertical_channels 4
7326#define STB_IMAGE_RESIZE_DO_VERTICALS
7327#define STB_IMAGE_RESIZE_VERTICAL_CONTINUE
7328#include STBIR__HEADER_FILENAME
7329
7330#define STBIR__vertical_channels 5
7331#define STB_IMAGE_RESIZE_DO_VERTICALS
7332#include STBIR__HEADER_FILENAME
7333
7334#define STBIR__vertical_channels 5
7335#define STB_IMAGE_RESIZE_DO_VERTICALS
7336#define STB_IMAGE_RESIZE_VERTICAL_CONTINUE
7337#include STBIR__HEADER_FILENAME
7338
7339#define STBIR__vertical_channels 6
7340#define STB_IMAGE_RESIZE_DO_VERTICALS
7341#include STBIR__HEADER_FILENAME
7342
7343#define STBIR__vertical_channels 6
7344#define STB_IMAGE_RESIZE_DO_VERTICALS
7345#define STB_IMAGE_RESIZE_VERTICAL_CONTINUE
7346#include STBIR__HEADER_FILENAME
7347
7348#define STBIR__vertical_channels 7
7349#define STB_IMAGE_RESIZE_DO_VERTICALS
7350#include STBIR__HEADER_FILENAME
7351
7352#define STBIR__vertical_channels 7
7353#define STB_IMAGE_RESIZE_DO_VERTICALS
7354#define STB_IMAGE_RESIZE_VERTICAL_CONTINUE
7355#include STBIR__HEADER_FILENAME
7356
7357#define STBIR__vertical_channels 8
7358#define STB_IMAGE_RESIZE_DO_VERTICALS
7359#include STBIR__HEADER_FILENAME
7360
7361#define STBIR__vertical_channels 8
7362#define STB_IMAGE_RESIZE_DO_VERTICALS
7363#define STB_IMAGE_RESIZE_VERTICAL_CONTINUE
7364#include STBIR__HEADER_FILENAME
7365
7366typedef void STBIR_VERTICAL_GATHERFUNC( float * output, float const * coeffs, float const ** inputs, float const * input0_end );
7367
7368static STBIR_VERTICAL_GATHERFUNC * stbir__vertical_gathers[ 8 ] =
7369{
7370 stbir__vertical_gather_with_1_coeffs,stbir__vertical_gather_with_2_coeffs,stbir__vertical_gather_with_3_coeffs,stbir__vertical_gather_with_4_coeffs,stbir__vertical_gather_with_5_coeffs,stbir__vertical_gather_with_6_coeffs,stbir__vertical_gather_with_7_coeffs,stbir__vertical_gather_with_8_coeffs
7371};
7372
7373static STBIR_VERTICAL_GATHERFUNC * stbir__vertical_gathers_continues[ 8 ] =
7374{
7375 stbir__vertical_gather_with_1_coeffs_cont,stbir__vertical_gather_with_2_coeffs_cont,stbir__vertical_gather_with_3_coeffs_cont,stbir__vertical_gather_with_4_coeffs_cont,stbir__vertical_gather_with_5_coeffs_cont,stbir__vertical_gather_with_6_coeffs_cont,stbir__vertical_gather_with_7_coeffs_cont,stbir__vertical_gather_with_8_coeffs_cont
7376};
7377
7378typedef void STBIR_VERTICAL_SCATTERFUNC( float ** outputs, float const * coeffs, float const * input, float const * input_end );
7379
7380static STBIR_VERTICAL_SCATTERFUNC * stbir__vertical_scatter_sets[ 8 ] =
7381{
7382 stbir__vertical_scatter_with_1_coeffs,stbir__vertical_scatter_with_2_coeffs,stbir__vertical_scatter_with_3_coeffs,stbir__vertical_scatter_with_4_coeffs,stbir__vertical_scatter_with_5_coeffs,stbir__vertical_scatter_with_6_coeffs,stbir__vertical_scatter_with_7_coeffs,stbir__vertical_scatter_with_8_coeffs
7383};
7384
7385static STBIR_VERTICAL_SCATTERFUNC * stbir__vertical_scatter_blends[ 8 ] =
7386{
7387 stbir__vertical_scatter_with_1_coeffs_cont,stbir__vertical_scatter_with_2_coeffs_cont,stbir__vertical_scatter_with_3_coeffs_cont,stbir__vertical_scatter_with_4_coeffs_cont,stbir__vertical_scatter_with_5_coeffs_cont,stbir__vertical_scatter_with_6_coeffs_cont,stbir__vertical_scatter_with_7_coeffs_cont,stbir__vertical_scatter_with_8_coeffs_cont
7388};
7389
7390
7391// pointer to an output pixel, relative to the output region (offset_x/offset_y are already folded into output_data)
7392static void * stbir__output_pixel_ptr( stbir__info const * stbir_info, int x, int y )
7393{
7394 return ( (char*) stbir_info->output_data ) + (ptrdiff_t)( y * stbir_info->output_stride_bytes ) + ( (size_t)x * (size_t)stbir_info->channels * (size_t)stbir__type_size[ stbir_info->output_type ] );
7395}
7396
7397static void stbir__encode_scanline( stbir__info const * stbir_info, float * encode_buffer, int row, stbir__column_slice const * slice STBIR_ONLY_PROFILE_GET_SPLIT_INFO )
7398{
7399 int num_pixels = slice->width;
7400 int channels = stbir_info->channels;
7401 ptrdiff_t width_times_channels = (ptrdiff_t)num_pixels * channels;
7402 void * output_buffer_data = stbir__output_pixel_ptr( stbir_info, slice->x0, row );
7403 void * output_buffer;
7404
7405 // un-alpha weight if we need to
7406 if ( stbir_info->alpha_unweight )
7407 {
7408 STBIR_PROFILE_START( unalpha );
7409 stbir_info->alpha_unweight( encode_buffer, width_times_channels );
7410 STBIR_PROFILE_END( unalpha );
7411 }
7412
7413 // write directly into output by default
7414 output_buffer = output_buffer_data;
7415
7416 // if we have an output callback, we first convert the decode buffer in place (and then hand that to the callback)
7417 if ( stbir_info->out_pixels_cb )
7418 output_buffer = encode_buffer;
7419
7420 STBIR_PROFILE_START( encode );
7421 // convert into the output buffer
7422 stbir_info->encode_pixels( output_buffer, width_times_channels, encode_buffer );
7423 STBIR_PROFILE_END( encode );
7424
7425 // if we have an output callback, call it to send the data (x and y are in full output image space, not region relative)
7426 if ( stbir_info->out_pixels_cb )
7427 stbir_info->out_pixels_cb( output_buffer, num_pixels, stbir_info->offset_x + slice->x0, stbir_info->offset_y + row, stbir_info->user_data );
7428}
7429
7430
7431// Get the ring buffer pointer for an index
7432static float* stbir__get_ring_buffer_entry(stbir__info const * stbir_info, stbir__per_split_info const * split_info, int index )
7433{
7434 STBIR_ASSERT( index < stbir_info->ring_buffer_num_entries );
7435
7436 #ifdef STBIR__SEPARATE_ALLOCATIONS
7437 return split_info->ring_buffers[ index ];
7438 #else
7439 return (float*) ( ( (char*) split_info->ring_buffer ) + ( (size_t) index * stbir_info->ring_buffer_length_bytes ) );
7440 #endif
7441}
7442
7443// Get the specified scan line from the ring buffer
7444static float* stbir__get_ring_buffer_scanline(stbir__info const * stbir_info, stbir__per_split_info const * split_info, int get_scanline)
7445{
7446 int ring_buffer_index = (split_info->ring_buffer_begin_index + (get_scanline - split_info->ring_buffer_first_scanline)) % stbir_info->ring_buffer_num_entries;
7447 return stbir__get_ring_buffer_entry( stbir_info, split_info, ring_buffer_index );
7448}
7449
7450// the horizontal pass over one slice's output pixels.
7451static void stbir__resample_horizontal_gather(stbir__info const * stbir_info, float* output_buffer, float const * input_buffer, stbir__column_slice const * slice STBIR_ONLY_PROFILE_GET_SPLIT_INFO )
7452{
7453 stbir__sampler const * horizontal = &stbir_info->horizontal;
7454 int channels = stbir_info->effective_channels;
7455 int point_copy = ( horizontal->filter_enum == STBIR_FILTER_POINT_SAMPLE ) && ( stbir__scale_is_copy( &horizontal->scale_info ) );
7456 int prologue = horizontal->shared_prologue;
7457 int period = horizontal->shared_period;
7458 int epilogue_start = horizontal->num_contributors - horizontal->shared_epilogue;
7459 int x = slice->x0, x_end = slice->x0 + slice->width;
7460 int n0 = slice->first_n0;
7461
7462 STBIR_PROFILE_START( horizontal );
7463 while ( x < x_end )
7464 {
7465 float const * decode = input_buffer + (ptrdiff_t)( n0 - slice->in_n0 ) * channels; // input_buffer holds input pixel in_n0 at index zero
7466 int run_end = x_end;
7467 int entry_stride = horizontal->entry_floats * (int) sizeof( float );
7468
7469 if ( ( period ) && ( x < epilogue_start ) )
7470 {
7471 if ( period == 1 )
7472 {
7473 // one set for the whole periodic region: the prologue is its own run, then the region is one run over that set
7474 run_end = ( x < prologue ) ? prologue : epilogue_start;
7475 if ( x >= prologue )
7476 entry_stride = 0;
7477 }
7478 else
7479 {
7480 // to the end of this pass over the block (from the prologue, the first pass), but not into the epilogue
7481 int pass = ( x < prologue ) ? 0 : ( ( x - prologue ) / period );
7482 run_end = prologue + ( pass + 1 ) * period;
7483 if ( run_end > epilogue_start )
7484 run_end = epilogue_start;
7485 }
7486 if ( run_end > x_end )
7487 run_end = x_end;
7488 }
7489
7490 if ( point_copy )
7491 STBIR_MEMCPY( output_buffer, decode, ( run_end - x ) * sizeof( float ) * channels );
7492 else if ( ( entry_stride == 0 ) && ( stbir_info->horizontal_gather_period1 ) )
7493 stbir_info->horizontal_gather_period1( output_buffer, run_end - x, decode, stbir__output_column_entry( horizontal, x ), 0 );
7494 else
7495 stbir_info->horizontal_gather_channels( output_buffer, run_end - x, decode, stbir__output_column_entry( horizontal, x ), entry_stride );
7496 output_buffer += (ptrdiff_t)( run_end - x ) * channels;
7497
7498 // where the next run starts reading: the epilogue, or a block boundary
7499 if ( run_end == epilogue_start )
7500 n0 = horizontal->shared_epilogue_n0;
7501 else if ( period )
7502 n0 = horizontal->shared_block_n0 + ( ( run_end - prologue ) / period ) * horizontal->shared_block_shift;
7503 x = run_end;
7504 }
7505 STBIR_PROFILE_END( horizontal );
7506}
7507
7508// how many floats wide the buffer that the vertical pass works on is (decoded input pixels when vertical
7509// first, horizontally resampled output pixels otherwise)
7510static ptrdiff_t stbir__vertical_width_times_channels( stbir__info const * stbir_info, stbir__column_slice const * slice )
7511{
7512 int width = ( stbir_info->vertical_first ) ? ( slice->in_n1 - slice->in_n0 + 1 ) : slice->width;
7513 return (ptrdiff_t)stbir_info->effective_channels * width;
7514}
7515
7516static void stbir__resample_vertical_gather(stbir__info const * stbir_info, stbir__per_split_info* split_info, int n, int contrib_n0, int contrib_n1, float const * vertical_coefficients, stbir__column_slice const * slice )
7517{
7518 float* encode_buffer = split_info->vertical_buffer;
7519 float* decode_buffer = split_info->decode_buffer;
7520 int vertical_first = stbir_info->vertical_first;
7521 ptrdiff_t width_times_channels = stbir__vertical_width_times_channels( stbir_info, slice );
7522
7523 STBIR_ASSERT( stbir_info->vertical.is_gather );
7524
7525 // loop over the contributing scanlines and scale into the buffer
7526 STBIR_PROFILE_START( vertical );
7527 {
7528 int k = 0, total = contrib_n1 - contrib_n0 + 1;
7529 STBIR_ASSERT( total > 0 );
7530 do {
7531 float const * inputs[8];
7532 int i, cnt = total;
7533 if ( cnt > 8 )
7534 cnt = 8;
7535 for( i = 0 ; i < cnt ; i++ )
7536 inputs[ i ] = stbir__get_ring_buffer_scanline(stbir_info, split_info, k+i+contrib_n0 );
7537
7538 // call the N scanlines at a time function (up to 8 scanlines of blending at once)
7539 ((k==0)?stbir__vertical_gathers:stbir__vertical_gathers_continues)[cnt-1]( (vertical_first) ? decode_buffer : encode_buffer, vertical_coefficients + k, inputs, inputs[0] + width_times_channels );
7540 k += cnt;
7541 total -= cnt;
7542 } while ( total );
7543 }
7544 STBIR_PROFILE_END( vertical );
7545
7546 if ( vertical_first )
7547 {
7548 // Now resample the gathered vertical data in the horizontal axis into the encode buffer
7549 decode_buffer[ width_times_channels ] = 0.0f; // clear two over for horizontals with a remnant of 3
7550 decode_buffer[ width_times_channels+1 ] = 0.0f;
7551 stbir__resample_horizontal_gather(stbir_info, encode_buffer, decode_buffer, slice STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
7552 }
7553
7554 stbir__encode_scanline( stbir_info, encode_buffer, n, slice STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
7555}
7556
7557static void stbir__decode_and_resample_for_vertical_gather_loop(stbir__info const * stbir_info, stbir__per_split_info* split_info, int n, stbir__column_slice const * slice)
7558{
7559 int ring_buffer_index;
7560 float* ring_buffer;
7561
7562 // Decode the nth scanline from the source image into the decode buffer.
7563 stbir__decode_scanline( stbir_info, n, split_info->decode_buffer, slice STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
7564
7565 // update new end scanline
7566 split_info->ring_buffer_last_scanline = n;
7567
7568 // get ring buffer
7569 ring_buffer_index = (split_info->ring_buffer_begin_index + (split_info->ring_buffer_last_scanline - split_info->ring_buffer_first_scanline)) % stbir_info->ring_buffer_num_entries;
7570 ring_buffer = stbir__get_ring_buffer_entry(stbir_info, split_info, ring_buffer_index);
7571
7572 // Now resample it into the ring buffer.
7573 stbir__resample_horizontal_gather( stbir_info, ring_buffer, split_info->decode_buffer, slice STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
7574
7575 // Now it's sitting in the ring buffer ready to be used as source for the vertical sampling.
7576}
7577
7578// A trimmed zero coefficient can make a row's range start later than the next row's ((4,4) then (3,6)), and the ring only
7579// loads forward. So look ahead up to max_look rows from y for the lowest first scanline; *out_row is which row had it.
7580static int stbir__lowest_scanline_needed( stbir__sampler const * vertical, int y, int max_look, int * out_row )
7581{
7582 int start_n0, lowest_n0, n0, n1, j;
7583
7584 stbir__output_range( vertical, y, &start_n0, &n1 );
7585 lowest_n0 = start_n0;
7586 *out_row = 0;
7587 for (j = 1; j <= max_look; j++)
7588 {
7589 stbir__output_range( vertical, y + j, &n0, &n1 );
7590 if ( n0 > start_n0 )
7591 break;
7592 if ( n0 < lowest_n0 )
7593 {
7594 lowest_n0 = n0;
7595 *out_row = j;
7596 }
7597 }
7598 return lowest_n0;
7599}
7600
7601static void stbir__vertical_gather_loop( stbir__info const * stbir_info, stbir__per_split_info* split_info, int split_count, stbir__column_slice const * slice )
7602{
7603 stbir__sampler const * vertical = &stbir_info->vertical;
7604 int y, start_output_y, end_output_y;
7605
7606 STBIR_ASSERT( vertical->is_gather );
7607
7608 start_output_y = split_info->start_output_y;
7609 end_output_y = split_info[split_count-1].end_output_y;
7610
7611 // initialize the ring buffer for gathering: start it at the lowest scanline the first rows need
7612 {
7613 int look = vertical->filter_pixel_margin * 3, unused;
7614 if ( look > ( end_output_y - start_output_y - 1 ) )
7615 look = end_output_y - start_output_y - 1;
7616 split_info->ring_buffer_begin_index = 0;
7617 split_info->ring_buffer_first_scanline = stbir__lowest_scanline_needed( vertical, start_output_y, look, &unused );
7618 split_info->ring_buffer_last_scanline = split_info->ring_buffer_first_scanline - 1; // means "empty"
7619 }
7620
7621 for (y = start_output_y; y < end_output_y; y++)
7622 {
7623 int in_first_scanline, in_last_scanline;
7624 float const * vertical_coefficients = stbir__output_coefficients( vertical, y );
7625
7626 stbir__output_range( vertical, y, &in_first_scanline, &in_last_scanline );
7627
7628 // rows with a zero weight at either end aren't worth decoding (the pack's shifted sets when the vertical shares the
7629 // horizontal's tables, kernels that are exactly zero at their support's edge)
7630 while ( ( in_first_scanline < in_last_scanline ) && ( vertical_coefficients[0] == 0.0f ) )
7631 {
7632 ++in_first_scanline;
7633 ++vertical_coefficients;
7634 }
7635 while ( ( in_last_scanline > in_first_scanline ) && ( vertical_coefficients[ in_last_scanline - in_first_scanline ] == 0.0f ) )
7636 --in_last_scanline;
7637
7638 // make sure the indexing hasn't broken
7639 STBIR_ASSERT( in_first_scanline >= split_info->ring_buffer_first_scanline );
7640
7641 // Load in new scanlines
7642 while (in_last_scanline > split_info->ring_buffer_last_scanline)
7643 {
7644 STBIR_ASSERT( ( split_info->ring_buffer_last_scanline - split_info->ring_buffer_first_scanline + 1 ) <= stbir_info->ring_buffer_num_entries );
7645
7646 // make sure there was room in the ring buffer when we add new scanlines
7647 if ( ( split_info->ring_buffer_last_scanline - split_info->ring_buffer_first_scanline + 1 ) == stbir_info->ring_buffer_num_entries )
7648 {
7649 split_info->ring_buffer_first_scanline++;
7650 split_info->ring_buffer_begin_index++;
7651 }
7652
7653 if ( stbir_info->vertical_first )
7654 {
7655 float * ring_buffer = stbir__get_ring_buffer_scanline( stbir_info, split_info, ++split_info->ring_buffer_last_scanline );
7656 // Decode the nth scanline from the source image into the decode buffer.
7657 stbir__decode_scanline( stbir_info, split_info->ring_buffer_last_scanline, ring_buffer, slice STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
7658 }
7659 else
7660 {
7661 stbir__decode_and_resample_for_vertical_gather_loop(stbir_info, split_info, split_info->ring_buffer_last_scanline + 1, slice);
7662 }
7663 }
7664
7665 // Now all buffers should be ready to write a row of vertical sampling, so do it.
7666 stbir__resample_vertical_gather(stbir_info, split_info, y, in_first_scanline, in_last_scanline, vertical_coefficients, slice );
7667 }
7668}
7669
7670#define STBIR__FLOAT_EMPTY_MARKER 3.0e+38F
7671#define STBIR__FLOAT_BUFFER_IS_EMPTY(ptr) ((ptr)[0]==STBIR__FLOAT_EMPTY_MARKER)
7672
7673static void stbir__encode_first_scanline_from_scatter(stbir__info const * stbir_info, stbir__per_split_info* split_info, stbir__column_slice const * slice)
7674{
7675 // evict a scanline out into the output buffer
7676 float* ring_buffer_entry = stbir__get_ring_buffer_entry(stbir_info, split_info, split_info->ring_buffer_begin_index );
7677
7678 // dump the scanline out
7679 stbir__encode_scanline( stbir_info, ring_buffer_entry, split_info->ring_buffer_first_scanline, slice STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
7680
7681 // mark it as empty
7682 ring_buffer_entry[ 0 ] = STBIR__FLOAT_EMPTY_MARKER;
7683
7684 // advance the first scanline
7685 split_info->ring_buffer_first_scanline++;
7686 if ( ++split_info->ring_buffer_begin_index == stbir_info->ring_buffer_num_entries )
7687 split_info->ring_buffer_begin_index = 0;
7688}
7689
7690static void stbir__horizontal_resample_and_encode_first_scanline_from_scatter(stbir__info const * stbir_info, stbir__per_split_info* split_info, stbir__column_slice const * slice)
7691{
7692 // evict a scanline out into the output buffer
7693
7694 float* ring_buffer_entry = stbir__get_ring_buffer_entry(stbir_info, split_info, split_info->ring_buffer_begin_index );
7695
7696 // Now resample it into the buffer.
7697 stbir__resample_horizontal_gather( stbir_info, split_info->vertical_buffer, ring_buffer_entry, slice STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
7698
7699 // dump the scanline out
7700 stbir__encode_scanline( stbir_info, split_info->vertical_buffer, split_info->ring_buffer_first_scanline, slice STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
7701
7702 // mark it as empty
7703 ring_buffer_entry[ 0 ] = STBIR__FLOAT_EMPTY_MARKER;
7704
7705 // advance the first scanline
7706 split_info->ring_buffer_first_scanline++;
7707 if ( ++split_info->ring_buffer_begin_index == stbir_info->ring_buffer_num_entries )
7708 split_info->ring_buffer_begin_index = 0;
7709}
7710
7711static void stbir__resample_vertical_scatter(stbir__info const * stbir_info, stbir__per_split_info* split_info, int n0, int n1, float const * vertical_coefficients, float const * vertical_buffer, float const * vertical_buffer_end )
7712{
7713 STBIR_ASSERT( !stbir_info->vertical.is_gather );
7714
7715 STBIR_PROFILE_START( vertical );
7716 {
7717 int k = 0, total = n1 - n0 + 1;
7718 STBIR_ASSERT( total > 0 );
7719 do {
7720 float * outputs[8];
7721 int i, n = total; if ( n > 8 ) n = 8;
7722 for( i = 0 ; i < n ; i++ )
7723 {
7724 outputs[ i ] = stbir__get_ring_buffer_scanline(stbir_info, split_info, k+i+n0 );
7725 if ( ( i ) && ( STBIR__FLOAT_BUFFER_IS_EMPTY( outputs[i] ) != STBIR__FLOAT_BUFFER_IS_EMPTY( outputs[0] ) ) ) // make sure runs are of the same type
7726 {
7727 n = i;
7728 break;
7729 }
7730 }
7731 // call the scatter to N scanlines at a time function (up to 8 scanlines of scattering at once)
7732 ((STBIR__FLOAT_BUFFER_IS_EMPTY( outputs[0] ))?stbir__vertical_scatter_sets:stbir__vertical_scatter_blends)[n-1]( outputs, vertical_coefficients + k, vertical_buffer, vertical_buffer_end );
7733 k += n;
7734 total -= n;
7735 } while ( total );
7736 }
7737
7738 STBIR_PROFILE_END( vertical );
7739}
7740
7741typedef void stbir__handle_scanline_for_scatter_func(stbir__info const * stbir_info, stbir__per_split_info* split_info, stbir__column_slice const * slice);
7742
7743static void stbir__vertical_scatter_loop( stbir__info const * stbir_info, stbir__per_split_info* split_info, int split_count, stbir__column_slice const * slice )
7744{
7745 int y, start_output_y, end_output_y, start_input_y, end_input_y;
7746 stbir__contributors* vertical_contributors = stbir_info->vertical.contributors;
7747 float const * vertical_coefficients = stbir_info->vertical.coefficients;
7748 stbir__handle_scanline_for_scatter_func * handle_scanline_for_scatter;
7749 void * scanline_scatter_buffer;
7750 void * scanline_scatter_buffer_end;
7751 int on_first_input_y, last_input_y;
7752 ptrdiff_t width_times_channels = stbir__vertical_width_times_channels( stbir_info, slice );
7753
7754 STBIR_ASSERT( !stbir_info->vertical.is_gather );
7755
7756 start_output_y = split_info->start_output_y;
7757 end_output_y = split_info[split_count-1].end_output_y; // may do multiple split counts
7758
7759 start_input_y = split_info->start_input_y;
7760 end_input_y = split_info[split_count-1].end_input_y;
7761
7762 // adjust for starting offset start_input_y
7763 y = start_input_y + stbir_info->vertical.filter_pixel_margin;
7764 vertical_contributors += y ;
7765 vertical_coefficients += (size_t)stbir_info->vertical.coefficient_width * y;
7766
7767 if ( stbir_info->vertical_first )
7768 {
7769 handle_scanline_for_scatter = stbir__horizontal_resample_and_encode_first_scanline_from_scatter;
7770 scanline_scatter_buffer = split_info->decode_buffer;
7771 }
7772 else
7773 {
7774 handle_scanline_for_scatter = stbir__encode_first_scanline_from_scatter;
7775 scanline_scatter_buffer = split_info->vertical_buffer;
7776 }
7777 scanline_scatter_buffer_end = ( (char*) scanline_scatter_buffer ) + sizeof( float ) * width_times_channels;
7778
7779 // initialize the ring buffer for scattering
7780 split_info->ring_buffer_first_scanline = start_output_y;
7781 split_info->ring_buffer_last_scanline = -1;
7782 split_info->ring_buffer_begin_index = -1;
7783
7784 // mark all the buffers as empty to start
7785 for( y = 0 ; y < stbir_info->ring_buffer_num_entries ; y++ )
7786 {
7787 float * decode_buffer = stbir__get_ring_buffer_entry( stbir_info, split_info, y );
7788 decode_buffer[ width_times_channels ] = 0.0f; // clear two over for horizontals with a remnant of 3
7789 decode_buffer[ width_times_channels+1 ] = 0.0f;
7790 decode_buffer[0] = STBIR__FLOAT_EMPTY_MARKER; // only used on scatter
7791 }
7792
7793 // do the loop in input space
7794 on_first_input_y = 1;
7795 last_input_y = start_input_y;
7796 for (y = start_input_y ; y < end_input_y; y++)
7797 {
7798 int out_first_scanline, out_last_scanline;
7799
7800 out_first_scanline = vertical_contributors->n0;
7801 out_last_scanline = vertical_contributors->n1;
7802
7803 // this input row is ours if the output rows it contributes to overlap the split's rows at all
7804 if ( ( out_last_scanline >= out_first_scanline ) && ( out_last_scanline >= start_output_y ) && ( out_first_scanline < end_output_y ) )
7805 {
7806 float const * coefficients = vertical_coefficients;
7807
7808 // keep track of the range actually seen for the next resize
7809 last_input_y = y;
7810 if ( ( on_first_input_y ) && ( y > start_input_y ) )
7811 split_info->start_input_y = y;
7812 on_first_input_y = 0;
7813
7814 // clip the region
7815 if ( out_first_scanline < start_output_y )
7816 {
7817 coefficients += start_output_y - out_first_scanline;
7818 out_first_scanline = start_output_y;
7819 }
7820
7821 if ( out_last_scanline >= end_output_y )
7822 out_last_scanline = end_output_y - 1;
7823
7824 STBIR_ASSERT( out_last_scanline - out_first_scanline + 1 <= stbir_info->ring_buffer_num_entries ); // (the ring is capped to the band's rows, so after the clip)
7825
7826 // if very first scanline, init the index
7827 if (split_info->ring_buffer_begin_index < 0)
7828 split_info->ring_buffer_begin_index = out_first_scanline - start_output_y;
7829
7830 STBIR_ASSERT( split_info->ring_buffer_begin_index <= out_first_scanline );
7831
7832 // Decode the nth scanline from the source image into the decode buffer.
7833 stbir__decode_scanline( stbir_info, y, split_info->decode_buffer, slice STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
7834
7835 // When horizontal first, we resample horizontally into the vertical buffer before we scatter it out
7836 if ( !stbir_info->vertical_first )
7837 stbir__resample_horizontal_gather( stbir_info, split_info->vertical_buffer, split_info->decode_buffer, slice STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
7838
7839 // Now it's sitting in the buffer ready to be distributed into the ring buffers.
7840
7841 // evict from the ring buffer if it is full and this row needs a new entry
7842 if ( ( ( split_info->ring_buffer_last_scanline - split_info->ring_buffer_first_scanline + 1 ) == stbir_info->ring_buffer_num_entries ) &&
7843 ( out_last_scanline > split_info->ring_buffer_last_scanline ) )
7844 handle_scanline_for_scatter( stbir_info, split_info, slice );
7845
7846 // Now the horizontal buffer is ready to write to all ring buffer rows, so do it.
7847 stbir__resample_vertical_scatter(stbir_info, split_info, out_first_scanline, out_last_scanline, coefficients, (float*)scanline_scatter_buffer, (float*)scanline_scatter_buffer_end );
7848
7849 // update the end of the buffer
7850 if ( out_last_scanline > split_info->ring_buffer_last_scanline )
7851 split_info->ring_buffer_last_scanline = out_last_scanline;
7852 }
7853 ++vertical_contributors;
7854 vertical_coefficients += stbir_info->vertical.coefficient_width;
7855 }
7856
7857 // now evict the scanlines that are left over in the ring buffer
7858 while ( split_info->ring_buffer_first_scanline < end_output_y )
7859 handle_scanline_for_scatter(stbir_info, split_info, slice);
7860
7861 // update the end_input_y if we do multiple resizes with the same data
7862 ++last_input_y;
7863 for( y = 0 ; y < split_count; y++ )
7864 if ( split_info[y].end_input_y > last_input_y )
7865 split_info[y].end_input_y = last_input_y;
7866}
7867
7868
7869static void stbir__set_sampler(stbir__sampler * samp, stbir_filter filter, stbir__kernel_callback_new * kernel_new, stbir__support_callback_new * support_new, stbir__radius_callback_new * radius, stbir__kernel_callback * kernel_old, stbir__support_callback * support_old, stbir_edge edge, stbir__scale_info * scale_info, int always_gather, void * user_data )
7870{
7871 // set filter
7872 if (filter == 0)
7873 {
7874 filter = STBIR_DEFAULT_FILTER_DOWNSAMPLE; // default to downsample
7875 if ( stbir__scale_is_upsample( scale_info ) )
7876 {
7877 if ( stbir__scale_is_copy( scale_info ) )
7878 filter = STBIR_FILTER_POINT_SAMPLE;
7879 else
7880 filter = STBIR_DEFAULT_FILTER_UPSAMPLE;
7881 }
7882 }
7883 samp->filter_enum = filter;
7884
7885 STBIR_ASSERT(samp->filter_enum != 0);
7886 STBIR_ASSERT((unsigned)samp->filter_enum < STBIR_FILTER_OTHER);
7887 samp->filter_kernel_new = stbir__builtin_kernels[ filter ];
7888 samp->filter_kernel_old = 0;
7889 samp->filter_support_new = stbir__builtin_supports[ filter ];
7890 samp->filter_radius = stbir__builtin_radii[ filter ];
7891 samp->filter_support_old = 0;
7892
7893 // a user filter is a kernel with a support or a radius (stbir_set_filter_callbacks_new refuses both), or an old
7894 // float kernel with an old float support (stbir_set_filter_callbacks)
7895 if ( ( kernel_new && ( support_new || radius ) ) || ( kernel_old && support_old ) )
7896 {
7897 samp->filter_kernel_new = kernel_new;
7898 samp->filter_kernel_old = kernel_old;
7899 samp->filter_support_new = support_new;
7900 samp->filter_radius = radius;
7901 samp->filter_support_old = support_old;
7902 samp->filter_enum = STBIR_FILTER_OTHER;
7903 }
7904
7905 samp->edge = edge;
7906 samp->filter_pixel_width = stbir__get_filter_pixel_width( samp->filter_radius, samp->filter_support_new, samp->filter_support_old, scale_info, user_data );
7907 // Gather is always better, but in extreme downsamples, you have to most or all of the data in memory
7908 // For horizontal, we always have all the pixels, so we always use gather here (always_gather==1).
7909 // For vertical, we use gather if scaling up (which means we will have samp->filter_pixel_width
7910 // scanlines in memory at once).
7911 samp->is_gather = 0;
7912 if ( stbir__scale_is_upsample( scale_info ) )
7913 samp->is_gather = 1;
7914 else if ( ( always_gather ) || ( samp->filter_pixel_width <= STBIR_FORCE_GATHER_FILTER_SCANLINES_AMOUNT ) )
7915 samp->is_gather = 2;
7916
7917 // pre calculate stuff based on the above
7918 samp->coefficient_width = stbir__get_coefficient_width(samp, samp->is_gather, user_data);
7919
7920 // filter_pixel_width is the conservative size in pixels of input that affect an output pixel.
7921 // In rare cases (only with 2 pix to 1 pix with the default filters), it's possible that the
7922 // filter will extend before or after the scanline beyond just one extra entire copy of the
7923 // scanline (we would hit the edge twice). We don't let you do that, so we clamp the total
7924 // width to 3x the total of input pixel (once for the scanline, once for the left side
7925 // overhang, and once for the right side). We only do this for edge mode, since the other
7926 // modes can just re-edge clamp back in again.
7927 if ( edge == STBIR_EDGE_WRAP )
7928 if ( ( samp->filter_pixel_width / 3 ) > scale_info->input_full_size )
7929 samp->filter_pixel_width = scale_info->input_full_size * 3;
7930
7931 // This is how much to expand buffers to account for filters seeking outside
7932 // the image boundaries.
7933 samp->filter_pixel_margin = samp->filter_pixel_width / 2;
7934
7935 // filter_pixel_margin is the amount that this filter can overhang on just one side of either
7936 // end of the scanline (left or the right). Since we only allow you to overhang 1 scanline's
7937 // worth of pixels, we clamp this one side of overhang to the input scanline size. Again,
7938 // this clamping only happens in rare cases with the default filters (2 pix to 1 pix).
7939 if ( edge == STBIR_EDGE_WRAP )
7940 if ( samp->filter_pixel_margin > scale_info->input_full_size )
7941 samp->filter_pixel_margin = scale_info->input_full_size;
7942
7943 samp->num_contributors = stbir__get_contributors(samp, samp->is_gather);
7944
7945 samp->entry_floats = 0;
7946 samp->shared_period = 0;
7947 samp->shared_prologue = 0;
7948 samp->shared_epilogue = 0;
7949 samp->contributors_size = (size_t)samp->num_contributors * sizeof(stbir__contributors);
7950 samp->coefficients_size = (size_t)samp->num_contributors * STBIR__COLUMN_ENTRY_FLOATS( samp->coefficient_width ) * sizeof(float) + sizeof(float)*STBIR_INPUT_CALLBACK_PADDING; // room for the horizontal's column entry ints, plus padding
7951}
7952
7953// Walks input pixels from start toward end (step is -1 or 1) and returns the last one that contributes to any output,
7954// or fallback if none did. When the output region is only a pixel or two wide the starting guess can be past the
7955// whole region, so a non contributing pixel only stops the walk once contributing ones have been seen.
7956static int stbir__last_contributing_input( int start, int end, int step, int fallback, stbir_real in_pixels_radius, stbir_real support, stbir__scale_info const * scale_info )
7957{
7958 int n = start;
7959 int last = fallback;
7960 int seen = 0;
7961
7962 while ( ( step < 0 ) ? ( n >= end ) : ( n <= end ) )
7963 {
7964 int out_first_pixel, out_last_pixel;
7965 stbir__input_out_pixel_range( scale_info, n, in_pixels_radius, support, &out_first_pixel, &out_last_pixel );
7966 if ( out_first_pixel > out_last_pixel )
7967 {
7968 if ( seen )
7969 break;
7970 }
7971 else
7972 {
7973 last = n;
7974 seen = 1;
7975 }
7976 n += step;
7977 }
7978 return last;
7979}
7980
7981static void stbir__get_conservative_extents( stbir__sampler * samp, stbir__contributors * range, void * user_data )
7982{
7983 stbir_real scale = samp->scale_info.scale;
7984 stbir_real in_pixels_radius = stbir__filter_radius_in_input_pixels( samp->filter_radius, samp->filter_support_new, samp->filter_support_old, &samp->scale_info, user_data );
7985 int input_full_size = samp->scale_info.input_full_size;
7986 stbir_edge edge = samp->edge;
7987 stbir_real inv_scale = samp->scale_info.inv_scale;
7988
7989 STBIR_ASSERT( samp->is_gather != 0 );
7990
7991 if ( samp->is_gather == 1 )
7992 {
7993 int in_first_pixel, in_last_pixel;
7994 stbir_real filter_support = in_pixels_radius;
7995 stbir_real out_filter_radius = filter_support * scale;
7996
7997 stbir__output_in_pixel_range( &samp->scale_info, 0, out_filter_radius, filter_support, edge, &in_first_pixel, &in_last_pixel );
7998 range->n0 = in_first_pixel;
7999 stbir__output_in_pixel_range( &samp->scale_info, samp->scale_info.output_sub_size - 1, out_filter_radius, filter_support, edge, &in_first_pixel, &in_last_pixel );
8000 range->n1 = in_last_pixel;
8001 }
8002 else if ( samp->is_gather == 2 ) // downsample gather, refine
8003 {
8004 stbir_real filter_support = in_pixels_radius / inv_scale; // in output pixels
8005 int filter_pixel_margin = samp->filter_pixel_margin;
8006 int output_sub_size = samp->scale_info.output_sub_size;
8007 int in_first_pixel, in_last_pixel;
8008
8009 // get a conservative area of the input range
8010 stbir__calculate_in_pixel_range( &in_first_pixel, &in_last_pixel, (stbir_real)0.0, (stbir_real)0.0, samp->scale_info.inv_scale, samp->scale_info.pixel_shift, input_full_size, edge );
8011 range->n0 = in_first_pixel;
8012 stbir__calculate_in_pixel_range( &in_first_pixel, &in_last_pixel, (stbir_real)output_sub_size, (stbir_real)0.0, samp->scale_info.inv_scale, samp->scale_info.pixel_shift, input_full_size, edge );
8013 range->n1 = in_last_pixel;
8014
8015 // walk down through the margin from the start of the area to find the bottom, and up from its end to find the top
8016 // (the coefficient generator never goes past input_full_size + filter_pixel_margin - 1)
8017 range->n0 = stbir__last_contributing_input( range->n0 + 1, -filter_pixel_margin, -1, range->n0, in_pixels_radius, filter_support, &samp->scale_info );
8018 range->n1 = stbir__last_contributing_input( range->n1 - 1, input_full_size + filter_pixel_margin - 1, 1, range->n1, in_pixels_radius, filter_support, &samp->scale_info );
8019 }
8020
8021 // when polyphase, the coefficient generator only calculates the first numerator outputs and then copies them
8022 // forward shifted by denominator (exact integer math). the top of the range above was calculated directly at the
8023 // far end of the image in floating point, which can round the other way, so widen by one - and when the fraction is
8024 // an approximation of the scale, the copies drift from the real positions by up to approximated_drift input pixels.
8025 if ( ( samp->scale_info.scale_is_rational ) && ( (int)samp->scale_info.scale_numerator < samp->scale_info.output_sub_size ) )
8026 range->n1 += 1 + samp->scale_info.approximated_drift;
8027
8028 // with WRAP the range is allowed to hang off either edge (those pixels get decoded from the other end of
8029 // the scanline), the other modes fold everything back into the scanline so clamp them to it
8030 if ( samp->edge != STBIR_EDGE_WRAP )
8031 {
8032 // for reflect, the margin taps get folded back into the image, and on tiny inputs we
8033 // land beyond, so widen to include them
8034 if ( samp->edge == STBIR_EDGE_REFLECT )
8035 {
8036 if ( range->n0 < 0 )
8037 range->n1 = stbir__max( range->n1, stbir__min( -range->n0, input_full_size - 1 ) );
8038 if ( range->n1 >= input_full_size )
8039 range->n0 = stbir__min( range->n0, stbir__max( 0, ( input_full_size - 1 ) - ( range->n1 - input_full_size ) ) ); // ( 2 x size - n1 - 1, written so it can't overflow at 2^30 )
8040 }
8041
8042 // for non-edge-wrap modes, we never read over the edge, so clamp
8043 if ( range->n0 < 0 )
8044 range->n0 = 0;
8045 if ( range->n1 >= input_full_size )
8046 range->n1 = input_full_size - 1;
8047 }
8048}
8049
8050static void stbir__get_split_info( stbir__per_split_info* split_info, int splits, int output_height, int vertical_pixel_margin, int input_full_height, int is_gather, stbir__sampler const * vertical )
8051{
8052 int start_y = 0;
8053 int remaining = output_height;
8054 int i;
8055
8056 for (i = 0; i < splits; i++)
8057 {
8058 int rows = remaining / ( splits - i );
8059
8060 split_info[i].start_output_y = start_y;
8061 split_info[i].end_output_y = start_y + rows;
8062
8063 // Don't start a split on a row whose range starts later than a following row's (see stbir__lowest_scanline_needed):
8064 // move the rows up to that one into the previous split, so it ends with the unusual row instead of this one
8065 // starting on it. Only this split's rows are looked at (the next split's first row is its own problem).
8066 if ( ( is_gather ) && ( i ) && ( rows > 1 ) )
8067 {
8068 int look = vertical_pixel_margin * 3;
8069 int move;
8070
8071 if ( look > ( rows - 1 ) )
8072 look = ( rows - 1 );
8073 stbir__lowest_scanline_needed( vertical, start_y, look, &move );
8074 split_info[i-1].end_output_y += move;
8075 split_info[i].start_output_y += move;
8076 }
8077
8078 start_y += rows;
8079 remaining -= rows;
8080
8081 // scatter range (updated to minimum as you run it)
8082 split_info[i].start_input_y = -vertical_pixel_margin;
8083 split_info[i].end_input_y = input_full_height + vertical_pixel_margin;
8084 }
8085}
8086
8087static void stbir__free_internal_mem( stbir__info *info )
8088{
8089 #define STBIR__FREE_AND_CLEAR( ptr ) { if ( ptr ) { void * p = (ptr); (ptr) = 0; STBIR_FREE( p, info->user_data); } }
8090
8091 if ( info )
8092 {
8093 #ifndef STBIR__SEPARATE_ALLOCATIONS
8094 STBIR__FREE_AND_CLEAR( info->alloced_mem );
8095 #else
8096 // a build whose allocation failed part way comes through here too: info and its pointer arrays are zeroed when
8097 // allocated, so every pointer is either an allocation or null
8098 int i,j;
8099
8100 if ( info->split_info )
8101 {
8102 for( i = 0 ; i < info->splits ; i++ )
8103 {
8104 stbir__per_split_info * split = info->split_info + i;
8105
8106 if ( split->ring_buffers )
8107 {
8108 for( j = 0 ; j < info->alloc_ring_buffer_num_entries ; j++ )
8109 {
8110 float * ring_buffer = split->ring_buffers[j];
8111 if ( ring_buffer )
8112 {
8113 #ifdef STBIR_SIMD8
8114 if ( info->effective_channels == 3 )
8115 --ring_buffer; // avx in 3 channel mode needs one float at the start of the buffer
8116 #endif
8117 STBIR_FREE( ring_buffer, info->user_data );
8118 }
8119 }
8120 STBIR__FREE_AND_CLEAR( split->ring_buffers );
8121 }
8122
8123 if ( split->decode_buffer_alloc )
8124 {
8125 #ifdef STBIR_SIMD8
8126 if ( info->effective_channels == 3 )
8127 split->decode_buffer_alloc = ( (float*)split->decode_buffer_alloc ) - 1; // avx in 3 channel mode needs one float at the start of the buffer
8128 #endif
8129 STBIR__FREE_AND_CLEAR( split->decode_buffer_alloc );
8130 }
8131 STBIR__FREE_AND_CLEAR( split->vertical_buffer_alloc );
8132 }
8133 STBIR__FREE_AND_CLEAR( info->split_info );
8134 }
8135 STBIR__FREE_AND_CLEAR( info->column_slices );
8136 if ( info->vertical.contributors != info->horizontal.contributors ) // (when shared, the vertical's coefficients point two ints into the horizontal's)
8137 {
8138 STBIR__FREE_AND_CLEAR( info->vertical.coefficients );
8139 STBIR__FREE_AND_CLEAR( info->vertical.contributors );
8140 }
8141 STBIR__FREE_AND_CLEAR( info->horizontal.coefficients );
8142 STBIR__FREE_AND_CLEAR( info->horizontal.contributors );
8143 STBIR__FREE_AND_CLEAR( info->alloced_mem );
8144 STBIR_FREE( info, info->user_data );
8145 #endif
8146 }
8147
8148 #undef STBIR__FREE_AND_CLEAR
8149}
8150
8151static int stbir__get_max_split( int splits, int height )
8152{
8153 int i;
8154 int max = 0;
8155
8156 for( i = 0 ; i < splits ; i++ )
8157 {
8158 int each = height / ( splits - i );
8159 if ( each > max )
8160 max = each;
8161 height -= each;
8162 }
8163 return max;
8164}
8165
8166static stbir__horizontal_gather_channels_func ** stbir__horizontal_gather_n_coeffs_funcs[8] =
8167{
8168 0, stbir__horizontal_gather_1_channels_with_n_coeffs_funcs, stbir__horizontal_gather_2_channels_with_n_coeffs_funcs, stbir__horizontal_gather_3_channels_with_n_coeffs_funcs, stbir__horizontal_gather_4_channels_with_n_coeffs_funcs, 0,0, stbir__horizontal_gather_7_channels_with_n_coeffs_funcs
8169};
8170
8171static stbir__horizontal_gather_channels_func ** stbir__horizontal_gather_channels_funcs[8] =
8172{
8173 0, stbir__horizontal_gather_1_channels_funcs, stbir__horizontal_gather_2_channels_funcs, stbir__horizontal_gather_3_channels_funcs, stbir__horizontal_gather_4_channels_funcs, 0,0, stbir__horizontal_gather_7_channels_funcs
8174};
8175
8176// the polyphase period 1 kernels, [channels][coefficient count] (see the horizontal kernel section)
8177static stbir__horizontal_gather_channels_func ** stbir__horizontal_gather_period1_funcs[8] =
8178{
8179 0, stbir__horizontal_gather_1_channels_period1_funcs, stbir__horizontal_gather_2_channels_period1_funcs, stbir__horizontal_gather_3_channels_period1_funcs, stbir__horizontal_gather_4_channels_period1_funcs, 0,0, stbir__horizontal_gather_7_channels_period1_funcs
8180};
8181
8182// debug overrides (the tests set these; they are statics, so per translation unit): 0 = off
8183static int stbir__force_vertical_first = 0; // 1 = horizontal first, 2 = vertical first (dotimings, refcheck -vfirst)
8184static int stbir__chose_vertical_first = 0; // (read only) what the cost model picked for the last build
8185static int stbir__split_mode = 0; // 1 = row bands only, 2 = column groups only (refcheck_threads -row_split / -colsplit)
8186static int stbir__split_force_cols = 0; // force this many column groups, the bands are splits / this (refcheck_threads -grid)
8187static int stbir__column_slices = 0; // force this many column slices instead of the L2 fit (refcheck_time cfg N:)
8188static int stbir__4K_vertical_dest_shift = 0; // bytes added to the vertical gather destination's page offset (refcheck_time -vshift)
8189
8190// Vertical first or horizontal first? It matters most when the two axes scale very differently (2x wider and half the
8191// height: the vertical first order is around 3x faster), and 20-40% in ordinary cases. The decision is a kernel cost
8192// model: both orders move the same decoded pixels, so the question is only how much each kernel charges per byte it
8193// streams and the multiplies each direction does (vertical is more SIMDable than horizontal). We calculate the costs as:
8194//
8195// vertical first: (taps_v + 1) / s_h * rv + (1/s_h + 1) * rh (vertical over input-width rows, horizontal over output rows)
8196// horizontal first: (1/s_v) * (1/s_h + 1) * rh + (taps_v + 1) * rv (horizontal over every input row, vertical over output-width rows)
8197// (1/s = input/output per axis, with WRAP's margin pixels and scanlines added to the input)
8198//
8199// rh and rv are the scale factors for the horizontal and vertical costs. We then measure a bunch of horizontal
8200// first and vertical first resizes on a bunch of machines to train small tables to give us a relative speed of
8201// each direction.
8202
8203// horizontal tap count -> table column: 1, 2, 3, 4, 5-6, 7-8, 9-12, 13-17, 18+
8204static int stbir__horizontal_cost_bucket( int taps )
8205{
8206 if ( taps <= 4 ) return ( taps < 1 ) ? 0 : taps - 1;
8207 if ( taps <= 6 ) return 4;
8208 if ( taps <= 8 ) return 5;
8209 if ( taps <= 12 ) return 6;
8210 if ( taps <= 17 ) return 7;
8211 return 8;
8212}
8213
8214// [1,2,3,4,7 channels][bucket]: horizontal kernel cost per byte streamed, relative to the vertical kernel's
8215static float stbir__horizontal_cost[5][9] =
8216{
8217 { 12.472f, 6.718f, 6.656f, 8.008f, 5.957f, 6.068f, 6.474f, 7.646f, 10.184f },
8218 { 8.067f, 4.192f, 5.091f, 4.906f, 4.816f, 5.584f, 5.741f, 6.971f, 6.182f },
8219 { 5.495f, 3.718f, 3.206f, 7.861f, 4.816f, 5.584f, 5.741f, 5.584f, 6.068f },
8220 { 3.651f, 2.135f, 2.791f, 3.583f, 2.950f, 3.583f, 4.154f, 4.192f, 4.861f },
8221 { 2.434f, 2.001f, 2.521f, 3.583f, 2.817f, 3.236f, 3.550f, 3.822f, 4.271f },
8222};
8223
8224// the same when the horizontal is an integer downsample (the polyphase period 1 kernels)
8225static float stbir__horizontal_cost_period1[5][9] =
8226{
8227 { 12.406f, 6.718f, 6.534f, 5.186f, 5.114f, 5.138f, 6.656f, 5.431f, 10.374f },
8228 { 8.001f, 4.154f, 3.583f, 4.003f, 3.276f, 4.473f, 3.683f, 4.231f, 6.013f },
8229 { 5.024f, 3.033f, 3.583f, 3.683f, 3.268f, 4.432f, 3.650f, 4.231f, 5.283f },
8230 { 2.976f, 2.076f, 2.665f, 2.452f, 2.354f, 2.766f, 2.715f, 3.327f, 4.684f },
8231 { 2.184f, 1.928f, 2.342f, 1.983f, 2.233f, 3.005f, 2.715f, 3.421f, 3.858f },
8232};
8233
8234// a x b as a signed 128 bit value, for the exact cost comparison below
8235static void stbir__mul_64x64( stbir_int64 a, stbir_int64 b, stbir_int64 * hi, stbir_uint64 * lo )
8236{
8237 stbir_uint64 ua = (stbir_uint64) a, ub = (stbir_uint64) b;
8238 stbir_uint64 a0 = ua & 0xffffffffu, a1 = ua >> 32, b0 = ub & 0xffffffffu, b1 = ub >> 32;
8239 stbir_uint64 p00 = a0 * b0, p01 = a0 * b1, p10 = a1 * b0, p11 = a1 * b1;
8240 stbir_uint64 mid = ( p00 >> 32 ) + ( p01 & 0xffffffffu ) + ( p10 & 0xffffffffu );
8241 stbir_uint64 uhi = p11 + ( p01 >> 32 ) + ( p10 >> 32 ) + ( mid >> 32 );
8242 *lo = ( mid << 32 ) | ( p00 & 0xffffffffu );
8243 if ( a < 0 ) uhi -= ub; // the unsigned product of the two's complements, corrected to the signed one
8244 if ( b < 0 ) uhi -= ua;
8245 *hi = (stbir_int64) uhi;
8246}
8247
8248// a x b <= c x d, exactly
8249static int stbir__products_le( stbir_int64 a, stbir_int64 b, stbir_int64 c, stbir_int64 d )
8250{
8251 stbir_int64 lhi, rhi;
8252 stbir_uint64 llo, rlo;
8253 stbir__mul_64x64( a, b, &lhi, &llo );
8254 stbir__mul_64x64( c, d, &rhi, &rlo );
8255 return ( lhi < rhi ) || ( ( lhi == rhi ) && ( llo <= rlo ) );
8256}
8257
8258// WRAP is the one edge mode that changes the amount of work: the decoded rows carry the filter's margin pixels on both
8259// ends and the vertical runs over margin scanlines, so the input is effectively taps wider / taller.
8260static int stbir__should_do_vertical_first( int channel_index, int horizontal_filter_pixel_width, int horizontal_input_size, int horizontal_output_size, stbir_edge horizontal_edge, int horizontal_is_period1,
8261 int vertical_filter_pixel_width, int vertical_input_size, int vertical_output_size, stbir_edge vertical_edge )
8262{
8263 stbir_int64 horizontal_input, vertical_input, horizontal_output, vertical_output, vertical_taps, cost_ratio;
8264 int bucket, vertical_first;
8265
8266 // the horizontal kernel's cost relative to the vertical's, in 1/1000ths
8267 bucket = stbir__horizontal_cost_bucket( horizontal_filter_pixel_width );
8268 cost_ratio = (stbir_int64) ( ( horizontal_is_period1 ? stbir__horizontal_cost_period1 : stbir__horizontal_cost )[ channel_index ][ bucket ] * 1000.0f + 0.5f );
8269
8270 // the inputs with their wrap margins
8271 horizontal_input = (stbir_int64) horizontal_input_size + ( ( horizontal_edge == STBIR_EDGE_WRAP ) ? horizontal_filter_pixel_width : 0 );
8272 vertical_input = (stbir_int64) vertical_input_size + ( ( vertical_edge == STBIR_EDGE_WRAP ) ? vertical_filter_pixel_width : 0 );
8273 horizontal_output = horizontal_output_size;
8274 vertical_output = vertical_output_size;
8275 vertical_taps = (stbir_int64) vertical_filter_pixel_width + 1;
8276
8277 vertical_first = stbir__products_le( vertical_taps * vertical_output, ( horizontal_input - horizontal_output ) * 1000,
8278 cost_ratio * ( horizontal_input + horizontal_output ), vertical_input - vertical_output );
8279
8280 // the test hook: report the choice, and let the tests force it
8281 stbir__chose_vertical_first = vertical_first;
8282 if ( stbir__force_vertical_first )
8283 vertical_first = ( stbir__force_vertical_first == 2 ) ? 1 : 0;
8284
8285 return vertical_first;
8286}
8287
8288// layout lookups - must match stbir_internal_pixel_layout
8289static unsigned char stbir__pixel_channels[] = {
8290 1,2,3,3,4, // 1ch, 2ch, rgb, bgr, 4ch
8291 4,4,4,4,2,2, // RGBA,BGRA,ARGB,ABGR,RA,AR
8292 4,4,4,4,2,2, // RGBA_PM,BGRA_PM,ARGB_PM,ABGR_PM,RA_PM,AR_PM
8293};
8294
8295// the internal pixel layout enums are in a different order, so we can easily do range comparisons of types
8296// the public pixel layout is ordered in a way that if you cast num_channels (1-4) to the enum, you get something sensible
8297static stbir_internal_pixel_layout stbir__pixel_layout_convert_public_to_internal[] = {
8298 STBIRI_BGR, STBIRI_1CHANNEL, STBIRI_2CHANNEL, STBIRI_RGB, STBIRI_RGBA,
8299 STBIRI_4CHANNEL, STBIRI_BGRA, STBIRI_ARGB, STBIRI_ABGR, STBIRI_RA, STBIRI_AR,
8300 STBIRI_RGBA_PM, STBIRI_BGRA_PM, STBIRI_ARGB_PM, STBIRI_ABGR_PM, STBIRI_RA_PM, STBIRI_AR_PM,
8301};
8302
8303// We need a good guess at the L2 size for the column sizing (CACHE SIZE in the docs). We get this size, in order of preference as:
8304// - stbir_l2_cache_size_override, if you set it to something non-zero before resizing (static, starts at 0, bytes)
8305// - the OS, if you define one of these before including this header (we don't include the headers for you):
8306// STBIR_ASK_WINDOWS_FOR_L2_SIZE - needs <windows.h> and <malloc.h> (GetLogicalProcessorInformationEx into an _alloca buffer)
8307// STBIR_ASK_LINUX_FOR_L2_SIZE - needs <unistd.h> (sysconf _SC_LEVEL2_CACHE_SIZE)
8308// STBIR_ASK_MACOS_FOR_L2_SIZE - needs <sys/sysctl.h> (sysctlbyname hw.l2cachesize)
8309// - cpuid on x86/x64 (leaf 0x80000006, the L2 size in KB - Intel and AMD both fill it in)
8310// - 256 KB if none of that applies or answers
8311// An L2 of 1 MB or more is clamped to 512 KB to keep total resize internal memory use down.
8312
8313static int stbir_l2_cache_size_override = 0; // bytes, 0 = detect
8314
8315#if ( defined( _M_X64 ) || defined( _M_IX86 ) || defined( __x86_64__ ) || defined( __i386__ ) ) && !defined( _M_ARM64EC )
8316 #define STBIR__L2_FROM_CPUID
8317 #ifdef _MSC_VER
8318 #include <intrin.h>
8319 #else
8320 #include <cpuid.h>
8321 #endif
8322#endif
8323
8324// (asked on every build - cpuid is a few hundred cycles, and no cache means nothing shared between threads building at once)
8325static int stbir__l2_cache_size( void )
8326{
8327 int bytes = 0;
8328
8329 if ( stbir_l2_cache_size_override > 0 )
8330 return stbir_l2_cache_size_override;
8331
8332 #if defined( STBIR_ASK_WINDOWS_FOR_L2_SIZE )
8333 {
8334 // the level 2 data or unified cache of the first processor group that reports one. the list is a few hundred
8335 // bytes per cache level per core, so it goes on the stack (a 64 core machine's is ~30 KB; past 256 KB fall through)
8336 DWORD len = 0;
8337 GetLogicalProcessorInformationEx( RelationCache, 0, &len );
8338 if ( ( len ) && ( len <= 256 * 1024 ) )
8339 {
8340 char * buf = (char*) _alloca( len );
8341 if ( GetLogicalProcessorInformationEx( RelationCache, (SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX*) buf, &len ) )
8342 {
8343 DWORD ofs = 0;
8344 while ( ofs < len )
8345 {
8346 SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX * info = (SYSTEM_LOGICAL_PROCESSOR_INFORMATION_EX*) ( buf + ofs );
8347 if ( ( info->Relationship == RelationCache ) && ( info->Cache.Level == 2 ) && ( ( info->Cache.Type == CacheData ) || ( info->Cache.Type == CacheUnified ) ) )
8348 {
8349 bytes = (int) info->Cache.CacheSize;
8350 break;
8351 }
8352 ofs += info->Size;
8353 }
8354 }
8355 }
8356 }
8357 #elif defined( STBIR_ASK_LINUX_FOR_L2_SIZE )
8358 {
8359 long v = sysconf( _SC_LEVEL2_CACHE_SIZE );
8360 if ( v > 0 )
8361 bytes = (int) v;
8362 }
8363 #elif defined( STBIR_ASK_MACOS_FOR_L2_SIZE )
8364 {
8365 stbir_uint64 v = 0;
8366 size_t vlen = sizeof( v );
8367 if ( ( sysctlbyname( "hw.l2cachesize", &v, &vlen, 0, 0 ) == 0 ) && ( v > 0 ) )
8368 bytes = (int) v;
8369 }
8370 #endif
8371
8372 #if defined( STBIR__L2_FROM_CPUID )
8373 if ( bytes <= 0 ) // no OS query, or it didn't answer
8374 {
8375 unsigned int regs[4] = { 0, 0, 0, 0 };
8376 #ifdef _MSC_VER
8377 __cpuid( (int*) regs, 0x80000000 );
8378 if ( regs[0] >= 0x80000006 )
8379 {
8380 __cpuid( (int*) regs, 0x80000006 );
8381 bytes = (int) ( regs[2] >> 16 ) * 1024; // ECX[31:16] = L2 size in KB
8382 }
8383 #else
8384 if ( __get_cpuid_max( 0x80000000, 0 ) >= 0x80000006 )
8385 {
8386 __get_cpuid( 0x80000006, ®s[0], ®s[1], ®s[2], ®s[3] );
8387 bytes = (int) ( regs[2] >> 16 ) * 1024;
8388 }
8389 #endif
8390 }
8391 #endif
8392
8393 if ( bytes <= 0 )
8394 bytes = 256 * 1024;
8395 else if ( bytes >= 1024 * 1024 )
8396 bytes = 512 * 1024;
8397
8398 return bytes;
8399}
8400
8401// The splits (what a caller threads over) are a grid of split_cols column groups x split_rows row bands. Any grid gives
8402// the same pixels and much the same memory (a split holds one slice's buffers either way), so stbir__choose_split_grid
8403// picks the fastest by estimate, for every count from the request down to one split:
8404// - a row band re-decodes the vertical filter's taps rows before its first output row, so short bands cost more.
8405// - column groups are free while they are the slices a single thread would run anyway (the L2 fit); each extra slice
8406// costs stbir_split_slice_row_cost columns of work per row. A group streams at least stbir_split_min_group_bytes
8407// of each input row; below stbir_split_group_full_bytes it pays stbir_split_group_cost_pct per (full / bytes - 1)
8408// per 8 threads, and narrow rows cost (threads / 8)^2 x (64 KB / input row bytes) / stbir_split_stream_cost_div.
8409// - a thread costs stbir_thread_start_cost of work to start and join, and the count is capped so that the work
8410// (output pixels x taps x channels / 4) is at least stbir_split_min_work x splits^2.
8411// - each count is scored by its largest split, and the last threads are dropped while they add less than
8412// stbir_split_min_thread_gain_pct of a thread's speed-up each over the best smaller count.
8413// So the build may return fewer splits than asked for (which was always allowed); stbir_split_extra_threads adds some back.
8414
8415// the split estimate's tuning (statics so an app can override them before resizing, like stbir_l2_cache_size_override;
8416// work units are output pixels x taps x channels / 4, roughly nanoseconds of resizing on a fast core)
8417static int stbir_split_min_group_bytes = 1024; // the least input row a column group streams (bytes)
8418static int stbir_split_group_full_bytes = 4096; // below this much input row a column group pays the narrow group cost:
8419static int stbir_split_group_cost_pct = 30; // this percent x ( full / bytes - 1 ) x ( threads / 8 ) of the work
8420static int stbir_split_stream_cost_div = 170; // narrow rows under many threads: ( threads / 8 )^2 x ( 64 KB / input row bytes ) / this, whatever the grid
8421static int stbir_split_slice_row_cost = 250; // input columns' worth of work per row for each slice beyond the L2 fit
8422static int stbir_thread_start_cost = 12000; // starting and joining one thread, in work units (a spinning pool ~1000, a thread per job 50000+)
8423static int stbir_split_min_thread_gain_pct = 30; // the last threads must each add this percent of a thread's worth of speed-up, or they are dropped
8424static int stbir_split_min_work = 20000; // the work must be at least this x splits^2
8425static int stbir_split_extra_threads = 0; // use this many more splits than the estimate settles on (never more than asked for)
8426#define STBIR__SPLIT_MAX_SPLITS 256 // the grid search's table
8427
8428
8429// slice boundaries are multiples of this many pixels (so the SIMD loops only run their remnants at the end of the row)
8430#define STBIR__SLICE_STEP 16
8431
8432// the most slices a row can have (each at least a step wide)
8433static int stbir__slice_max_count( int output_columns )
8434{
8435 int n = output_columns / STBIR__SLICE_STEP;
8436 return ( n < 1 ) ? 1 : n;
8437}
8438
8439
8440// The split grid for a request of *splits: cols x rows, and *splits becomes cols * rows (never more than asked for).
8441static void stbir__choose_split_grid( int * splits, int * out_cols, int * out_rows, int mode, int output_w, int output_h, int input_w, int input_pixel_bytes, int taps_v, float inv_scale_v, int l2_slices, int effective_channels )
8442{
8443 int max_splits = *splits;
8444 int requested_splits;
8445 int max_cols = stbir__slice_max_count( output_w ); // a column group needs at least one slice
8446 int best_cols = 1;
8447 int best_rows = 1;
8448
8449 if ( max_splits < 1 )
8450 max_splits = 1;
8451 if ( max_splits > STBIR__SPLIT_MAX_SPLITS )
8452 max_splits = STBIR__SPLIT_MAX_SPLITS;
8453 requested_splits = max_splits;
8454
8455 if ( stbir__split_force_cols > 0 )
8456 {
8457 // Debug: a forced grid.
8458 best_cols = ( stbir__split_force_cols < max_cols ) ? stbir__split_force_cols : max_cols;
8459 if ( best_cols > max_splits )
8460 best_cols = max_splits;
8461 best_rows = max_splits / best_cols;
8462 }
8463 else if ( mode == 1 )
8464 {
8465 // Row bands only.
8466 best_rows = ( max_splits < output_h ) ? max_splits : output_h;
8467 }
8468 else if ( mode == 2 )
8469 {
8470 // Column groups only.
8471 best_cols = ( max_splits < max_cols ) ? max_splits : max_cols;
8472 }
8473 else
8474 {
8475 // Everything in fixed point: times are in 1/65536ths of the single threaded work.
8476 stbir_uint64 work = (stbir_uint64)output_w * (stbir_uint64)output_h * (stbir_uint64)taps_v * (stbir_uint64)effective_channels / 4;
8477 stbir_uint64 band_warmup = (stbir_uint64)( (float)taps_v * 65536.0f / ( ( inv_scale_v > 1.0f ) ? inv_scale_v : 1.0f ) ); // what a band re-decodes before its first row, in output rows' worth of work
8478 stbir_uint64 row_bytes = (stbir_uint64)input_w * (stbir_uint64)input_pixel_bytes;
8479 stbir_uint64 estimate_of[ STBIR__SPLIT_MAX_SPLITS + 1 ];
8480 int cols_of[ STBIR__SPLIT_MAX_SPLITS + 1 ];
8481 int rows_of[ STBIR__SPLIT_MAX_SPLITS + 1 ];
8482 int count, cols;
8483
8484 if ( work < 1 )
8485 work = 1;
8486 if ( row_bytes < 1 )
8487 row_bytes = 1;
8488
8489 // The thread start/join cost grows with the count: the work must be at least min_work x splits^2.
8490 while ( ( max_splits > 1 ) && ( work < (stbir_uint64)stbir_split_min_work * (stbir_uint64)max_splits * (stbir_uint64)max_splits ) )
8491 --max_splits;
8492
8493 // The best grid for every count up to the request, by estimated time (the largest split's - it sets the pace).
8494 for (count = 1; count <= requested_splits; count++)
8495 {
8496 estimate_of[count] = ~(stbir_uint64)0;
8497 cols_of[count] = 1;
8498 rows_of[count] = 1;
8499
8500 for (cols = 1; cols <= count; cols++)
8501 {
8502 int rows, rows_per_band, extra_slices;
8503 stbir_uint64 group_bytes, narrow_cost, stream_cost, per_row_cost, estimate;
8504
8505 if ( count % cols )
8506 continue;
8507 if ( cols > max_cols )
8508 break;
8509
8510 // A column group streams its share of each input row, and never less than min_group_bytes of it.
8511 group_bytes = row_bytes / (stbir_uint64)cols;
8512 if ( group_bytes < 1 )
8513 group_bytes = 1;
8514 if ( ( cols > 1 ) && ( group_bytes < (stbir_uint64)stbir_split_min_group_bytes ) )
8515 break;
8516
8517 // Narrow groups cost group_cost_pct per ( full / bytes - 1 ) per 8 threads.
8518 narrow_cost = 0;
8519 if ( ( cols > 1 ) && ( group_bytes < (stbir_uint64)stbir_split_group_full_bytes ) )
8520 narrow_cost = ( ( (stbir_uint64)stbir_split_group_full_bytes * 65536 ) / group_bytes - 65536 ) * (stbir_uint64)stbir_split_group_cost_pct * (stbir_uint64)count / ( 100 * 8 );
8521
8522 // Narrow rows under many threads cost the same whatever the grid (every row is a fresh page, latency bound, and
8523 // the latency climbs with the thread count): the larger of this and the narrow group cost.
8524 stream_cost = ( (stbir_uint64)count * (stbir_uint64)count * 65536 * 65536 ) / ( (stbir_uint64)64 * (stbir_uint64)stbir_split_stream_cost_div * row_bytes );
8525 if ( stream_cost > narrow_cost )
8526 narrow_cost = stream_cost;
8527
8528 rows = count / cols;
8529 if ( rows > output_h )
8530 continue;
8531
8532 // The slowest split sets the time: the largest band (rows rounded up) doing the largest group's share of the
8533 // slices (the slice count rounded up to a multiple of the groups), against an even share of the work.
8534 rows_per_band = ( output_h + rows - 1 ) / rows;
8535 extra_slices = ( ( l2_slices + cols - 1 ) / cols ) * cols - l2_slices;
8536 per_row_cost = 65536 + band_warmup / (stbir_uint64)rows_per_band + ( (stbir_uint64)extra_slices * (stbir_uint64)stbir_split_slice_row_cost * 65536 ) / (stbir_uint64)input_w + narrow_cost;
8537 estimate = ( (stbir_uint64)rows_per_band * (stbir_uint64)rows ) * per_row_cost / ( (stbir_uint64)output_h * (stbir_uint64)count );
8538 estimate += ( (stbir_uint64)count * (stbir_uint64)stbir_thread_start_cost * 65536 ) / work; // plus the threads' own start/join cost
8539
8540 if ( estimate < estimate_of[count] )
8541 {
8542 estimate_of[count] = estimate;
8543 cols_of[count] = cols;
8544 rows_of[count] = rows;
8545 }
8546 }
8547 }
8548
8549 // From the request down, drop threads while they add less than min_thread_gain_pct of a thread's speed-up each
8550 // over the best smaller count (not the next one down: a prime count only has bad grids, which makes the count
8551 // above it look good).
8552 count = max_splits;
8553 while ( count > 1 )
8554 {
8555 stbir_uint64 speedup = ( estimate_of[1] * 256 ) / estimate_of[count];
8556 stbir_uint64 best_speedup = 0;
8557 int fewer, best_fewer = 1;
8558
8559 for (fewer = 1; fewer < count; fewer++)
8560 {
8561 stbir_uint64 speedup_fewer = ( estimate_of[1] * 256 ) / estimate_of[fewer];
8562 if ( speedup_fewer > best_speedup )
8563 {
8564 best_speedup = speedup_fewer;
8565 best_fewer = fewer;
8566 }
8567 }
8568
8569 if ( ( speedup > best_speedup ) && ( ( speedup - best_speedup ) * 100 >= (stbir_uint64)stbir_split_min_thread_gain_pct * 256 * (stbir_uint64)( count - best_fewer ) ) )
8570 break;
8571
8572 // The extra threads over the best smaller count weren't worth it: fall back to it and check again.
8573 count = best_fewer;
8574 }
8575
8576 // The app wants more than the estimate settled on.
8577 if ( stbir_split_extra_threads > 0 )
8578 {
8579 count += stbir_split_extra_threads;
8580 if ( count > requested_splits )
8581 count = requested_splits;
8582 // A count no grid fits (more bands than rows): the next one down.
8583 while ( ( count > 1 ) && ( estimate_of[count] == ~(stbir_uint64)0 ) )
8584 --count;
8585 }
8586
8587 best_cols = cols_of[count];
8588 best_rows = rows_of[count];
8589 }
8590
8591 if ( best_cols > max_cols )
8592 best_cols = max_cols;
8593 if ( best_rows > output_h )
8594 best_rows = output_h;
8595 if ( best_cols < 1 )
8596 best_cols = 1;
8597 if ( best_rows < 1 )
8598 best_rows = 1;
8599
8600 *out_cols = best_cols;
8601 *out_rows = best_rows;
8602 *splits = best_cols * best_rows;
8603}
8604
8605// where slice number slice of num_slices ends
8606static int stbir__slice_end( int output_columns, int num_slices, int slice )
8607{
8608 stbir_uint64 end;
8609
8610 if ( slice >= num_slices - 1 )
8611 return output_columns;
8612
8613 end = ( (stbir_uint64)( slice + 1 ) * (stbir_uint64)output_columns ) / (stbir_uint64)num_slices;
8614 return ( (int)end / STBIR__SLICE_STEP ) * STBIR__SLICE_STEP;
8615}
8616
8617// the widest slice there can be for a count (for sizing the per slice buffers): the average plus a step of rounding
8618static int stbir__slice_columns_for( int output_columns, int slices )
8619{
8620 int columns = ( output_columns + slices - 1 ) / slices + STBIR__SLICE_STEP;
8621 if ( columns >= output_columns )
8622 columns = output_columns;
8623 return columns;
8624}
8625
8626// the input pixels a slice decodes for that many columns (see the sizing in stbir__alloc_internal_mem_and_build_samplers)
8627static int stbir__slice_in_span_for( stbir__sampler const * horizontal, int slice_columns, int conservative_in_span )
8628{
8629 int span = (int)( slice_columns * horizontal->scale_info.inv_scale ) + 1 + horizontal->filter_pixel_width + 2 * horizontal->coefficient_width + 4;
8630 if ( ( slice_columns >= horizontal->scale_info.output_sub_size ) || ( span > conservative_in_span ) )
8631 span = conservative_in_span;
8632 return span;
8633}
8634
8635// the ring buffer for one slice: its entries are horizontal output rows (slice_columns wide), or decoded input rows
8636// (slice_in_span wide) when vertical first
8637static size_t stbir__ring_bytes_for( stbir__sampler const * horizontal, int slice_columns, int conservative_in_span, int effective_channels, int vertical_first, int ring_entries )
8638{
8639 int row_pixels = vertical_first ? stbir__slice_in_span_for( horizontal, slice_columns, conservative_in_span ) : slice_columns;
8640 return (size_t)ring_entries * ( (size_t)row_pixels * effective_channels * sizeof(float) + sizeof(float)*STBIR_INPUT_CALLBACK_PADDING );
8641}
8642
8643// pick the slice count: the fewest slices whose ring buffer fits the L2 budget (allowing an eighth over - a little
8644// spill costs less than another slice's re-decode), or the most slices possible (16 columns each) if none fit
8645static int stbir__choose_column_slices( stbir__sampler const * horizontal, int conservative_in_span, int effective_channels, int vertical_first, int ring_entries )
8646{
8647 int output_columns = horizontal->scale_info.output_sub_size;
8648 size_t budget = (size_t)stbir__l2_cache_size();
8649 size_t allowed = budget + budget / 8;
8650 int max_slices = stbir__slice_max_count( output_columns );
8651 int slices;
8652
8653 for (slices = 1; slices < max_slices; slices++)
8654 {
8655 int slice_columns = stbir__slice_columns_for( output_columns, slices );
8656 if ( stbir__ring_bytes_for( horizontal, slice_columns, conservative_in_span, effective_channels, vertical_first, ring_entries ) <= allowed )
8657 return slices;
8658 }
8659 return max_slices;
8660}
8661
8662// splits the output columns into num_column_slices balanced slices on step boundaries (stbir__slice_end), and works
8663// out the input range each slice has to decode: the lowest n0 through the furthest pixel any of its kernels read
8664// (stbir__pack_coefficients arranged for those reads to stay within the conservative range).
8665static void stbir__build_column_slices( stbir__info * info, int slice_columns, int in_span_bound )
8666{
8667 stbir__sampler const * horizontal = &info->horizontal;
8668 int output_columns = horizontal->scale_info.output_sub_size;
8669 int widest = horizontal->coefficient_width; // the coefficients are packed to widest by now
8670 int first_column, end_column;
8671 int slice_index = 0;
8672
8673 STBIR__UNUSED( in_span_bound ); // both only checked in asserts
8674 STBIR__UNUSED( slice_columns );
8675
8676 for (first_column = 0; first_column < output_columns; first_column = end_column, slice_index++)
8677 {
8678 stbir__column_slice * slice = info->column_slices + slice_index;
8679 int in_first = 0x7fffffff;
8680 int in_last = -0x7fffffff;
8681 int x, unused;
8682
8683 end_column = stbir__slice_end( output_columns, info->num_column_slices, slice_index );
8684 STBIR_ASSERT( ( end_column > first_column ) && ( end_column <= output_columns ) && ( ( end_column - first_column ) <= slice_columns ) );
8685
8686 // the lowest first pixel through the furthest pixel any of the slice's kernels read
8687 for (x = first_column; x < end_column; x++)
8688 {
8689 stbir__contributors contributor;
8690 int last_read;
8691
8692 stbir__output_range( horizontal, x, &contributor.n0, &contributor.n1 );
8693 last_read = contributor.n0 + stbir__horizontal_read_extent( &contributor, widest ) - 1;
8694 if ( contributor.n0 < in_first )
8695 in_first = contributor.n0;
8696 if ( last_read > in_last )
8697 in_last = last_read;
8698 }
8699
8700 STBIR_ASSERT( in_first >= info->conservative_extents.n0 );
8701 STBIR_ASSERT( in_last <= info->conservative_extents.n1 );
8702 STBIR_ASSERT( ( in_last - in_first + 1 ) <= in_span_bound ); // the buffers were sized from in_span_bound in the first allocation pass
8703
8704 slice->x0 = first_column;
8705 slice->width = end_column - first_column;
8706 slice->in_n0 = in_first;
8707 slice->in_n1 = in_last;
8708 stbir__output_range( horizontal, first_column, &slice->first_n0, &unused );
8709 }
8710
8711 STBIR_ASSERT( slice_index == info->num_column_slices );
8712}
8713
8714// split s is row band ( s / split_cols ) of column group ( s % split_cols ).
8715static void stbir__setup_split_grid( stbir__info * info, int split_cols, int split_rows )
8716{
8717 int r, c, next = 0;
8718
8719 info->split_cols = split_cols;
8720 info->split_rows = split_rows;
8721 stbir__get_split_info( info->split_info, split_rows, info->vertical.scale_info.output_sub_size, info->vertical.filter_pixel_margin, info->vertical.scale_info.input_full_size, info->vertical.is_gather, &info->vertical );
8722 for( r = split_rows - 1 ; r >= 0 ; r-- )
8723 {
8724 stbir__per_split_info band = info->split_info[r];
8725 for( c = split_cols - 1 ; c >= 0 ; c-- )
8726 {
8727 stbir__per_split_info * s = info->split_info + r * split_cols + c;
8728 s->start_output_y = band.start_output_y;
8729 s->end_output_y = band.end_output_y;
8730 s->start_input_y = band.start_input_y;
8731 s->end_input_y = band.end_input_y;
8732 }
8733 }
8734 for( c = 0 ; c < split_cols ; c++ )
8735 {
8736 int count = ( info->num_column_slices - next ) / ( split_cols - c );
8737 for( r = 0 ; r < split_rows ; r++ )
8738 {
8739 info->split_info[ r * split_cols + c ].first_column_slice = next;
8740 info->split_info[ r * split_cols + c ].num_column_slices = count;
8741 }
8742 next += count;
8743 }
8744}
8745
8746// the per split buffers get a page of slack, to be repositioned within it at resize time (stbir__4K_alias_skew_buffers)
8747#define STBIR__4K_ALIAS_SKEW_SLACK 4096
8748
8749// What kind of alpha weighting the layouts call for: 0 = none, 1 = weight in, leave premultiplied, 2 = fancy weight
8750// in and unweight out, 3 = premultiplied in, unweight out, 4 = simple weight in and unweight out. Fancy weighting
8751// carries extra channels through the filters (*effective_channels).
8752static int stbir__alpha_weighting_type( stbir__sampler const * horizontal, stbir__sampler const * vertical, stbir_internal_pixel_layout input_pixel_layout, stbir_internal_pixel_layout output_pixel_layout, int fast_alpha, int * effective_channels )
8753{
8754 int input_has_alpha = ( input_pixel_layout >= STBIRI_RGBA ) && ( input_pixel_layout <= STBIRI_AR );
8755 int input_premultiplied = ( input_pixel_layout >= STBIRI_RGBA_PM ) && ( input_pixel_layout <= STBIRI_AR_PM );
8756 int output_has_alpha = ( output_pixel_layout >= STBIRI_RGBA ) && ( output_pixel_layout <= STBIRI_AR );
8757 int output_premultiplied = ( output_pixel_layout >= STBIRI_RGBA_PM ) && ( output_pixel_layout <= STBIRI_AR_PM );
8758
8759 int point_sampled = ( horizontal->filter_enum == STBIR_FILTER_POINT_SAMPLE ) && ( vertical->filter_enum == STBIR_FILTER_POINT_SAMPLE );
8760
8761 if ( input_has_alpha && output_has_alpha )
8762 {
8763 static int fancy_alpha_effective_cnts[6] = { 7, 7, 7, 7, 3, 3 };
8764 if ( point_sampled ) // nothing is blended, so weighting in and unweighting out would be the identity
8765 return 0;
8766 if ( fast_alpha )
8767 return 4;
8768 *effective_channels = fancy_alpha_effective_cnts[ input_pixel_layout - STBIRI_RGBA ];
8769 return 2;
8770 }
8771 // (the conversions between premultiplied and not still happen when point sampling)
8772 if ( input_premultiplied && output_has_alpha )
8773 return 3;
8774 if ( input_has_alpha && output_premultiplied )
8775 return 1;
8776 return 0;
8777}
8778
8779// The alpha weight and unweight functions for the weighting type (and the 3 channel color flip, which rides the same path).
8780static void stbir__set_alpha_functions( stbir__info * info, int alpha_weighting_type, stbir_internal_pixel_layout input_pixel_layout, stbir_internal_pixel_layout output_pixel_layout, float horizontal_scale )
8781{
8782 static stbir__alpha_weight_func * fancy_alpha_weights[6] = { stbir__fancy_alpha_weight_4ch, stbir__fancy_alpha_weight_4ch, stbir__fancy_alpha_weight_4ch, stbir__fancy_alpha_weight_4ch, stbir__fancy_alpha_weight_2ch, stbir__fancy_alpha_weight_2ch };
8783 static stbir__alpha_unweight_func * fancy_alpha_unweights[6] = { stbir__fancy_alpha_unweight_4ch, stbir__fancy_alpha_unweight_4ch, stbir__fancy_alpha_unweight_4ch, stbir__fancy_alpha_unweight_4ch, stbir__fancy_alpha_unweight_2ch, stbir__fancy_alpha_unweight_2ch };
8784 static stbir__alpha_weight_func * simple_alpha_weights[6] = { stbir__simple_alpha_weight_4ch, stbir__simple_alpha_weight_4ch, stbir__simple_alpha_weight_4ch, stbir__simple_alpha_weight_4ch, stbir__simple_alpha_weight_2ch, stbir__simple_alpha_weight_2ch };
8785 static stbir__alpha_unweight_func * simple_alpha_unweights[6] = { stbir__simple_alpha_unweight_4ch, stbir__simple_alpha_unweight_4ch, stbir__simple_alpha_unweight_4ch, stbir__simple_alpha_unweight_4ch, stbir__simple_alpha_unweight_2ch, stbir__simple_alpha_unweight_2ch };
8786
8787 info->alpha_weight = 0;
8788 info->alpha_unweight = 0;
8789
8790 if ( alpha_weighting_type == 2 )
8791 {
8792 // high quality alpha multiplying on the way in, dividing on the way out
8793 info->alpha_weight = fancy_alpha_weights[ input_pixel_layout - STBIRI_RGBA ];
8794 info->alpha_unweight = fancy_alpha_unweights[ output_pixel_layout - STBIRI_RGBA ];
8795 }
8796 else if ( alpha_weighting_type == 4 )
8797 {
8798 // fast alpha multiplying on the way in, dividing on the way out
8799 info->alpha_weight = simple_alpha_weights[ input_pixel_layout - STBIRI_RGBA ];
8800 info->alpha_unweight = simple_alpha_unweights[ output_pixel_layout - STBIRI_RGBA ];
8801 }
8802 else if ( alpha_weighting_type == 1 )
8803 {
8804 // fast alpha on the way in, leave in premultiplied form on way out
8805 info->alpha_weight = simple_alpha_weights[ input_pixel_layout - STBIRI_RGBA ];
8806 }
8807 else if ( alpha_weighting_type == 3 )
8808 {
8809 // incoming is premultiplied, fast alpha dividing on the way out - non-premultiplied output
8810 info->alpha_unweight = simple_alpha_unweights[ output_pixel_layout - STBIRI_RGBA ];
8811 }
8812
8813 // handle 3-chan color flipping, using the alpha weight path
8814 if ( ( ( input_pixel_layout == STBIRI_RGB ) && ( output_pixel_layout == STBIRI_BGR ) ) ||
8815 ( ( input_pixel_layout == STBIRI_BGR ) && ( output_pixel_layout == STBIRI_RGB ) ) )
8816 {
8817 // do the flipping on the smaller of the two ends
8818 if ( horizontal_scale < 1.0f )
8819 info->alpha_unweight = stbir__simple_flip_3ch;
8820 else
8821 info->alpha_weight = stbir__simple_flip_3ch;
8822 }
8823}
8824
8825// Shared coefficients require identical geometry; inactive rational and exact-position fields may be uninitialized.
8826static int stbir__scale_info_matches( stbir__scale_info const * horizontal, stbir__scale_info const * vertical )
8827{
8828 if ( ( horizontal->input_full_size != vertical->input_full_size ) || ( horizontal->output_sub_size != vertical->output_sub_size ) ||
8829 ( horizontal->scale != vertical->scale ) || ( horizontal->inv_scale != vertical->inv_scale ) || // (the positions come from the reals)
8830 ( horizontal->pixel_shift != vertical->pixel_shift ) || ( horizontal->scale_is_rational != vertical->scale_is_rational ) ||
8831 ( horizontal->positions_exact != vertical->positions_exact ) )
8832 {
8833 return 0;
8834 }
8835
8836 if ( horizontal->scale_is_rational &&
8837 ( ( horizontal->scale_numerator != vertical->scale_numerator ) || ( horizontal->scale_denominator != vertical->scale_denominator ) ) )
8838 {
8839 return 0;
8840 }
8841
8842 if ( horizontal->positions_exact &&
8843 ( ( horizontal->shift_pixels != vertical->shift_pixels ) || ( horizontal->tap_scale_in != vertical->tap_scale_in ) ||
8844 ( horizontal->tap_scale_out != vertical->tap_scale_out ) ) )
8845 {
8846 return 0;
8847 }
8848
8849 return 1;
8850}
8851
8852// Are the two samplers identical? (happens a lot with mipmap generation) Then the vertical can share the horizontal's tables.
8853static int stbir__samplers_match( stbir__sampler const * horizontal, stbir__sampler const * vertical )
8854{
8855 return ( horizontal->filter_kernel_new == vertical->filter_kernel_new ) && ( horizontal->filter_kernel_old == vertical->filter_kernel_old ) && ( horizontal->filter_support_new == vertical->filter_support_new ) && ( horizontal->filter_radius == vertical->filter_radius ) && ( horizontal->filter_support_old == vertical->filter_support_old ) &&
8856 ( horizontal->edge == vertical->edge ) && ( horizontal->is_gather == vertical->is_gather ) &&
8857 stbir__scale_info_matches( &horizontal->scale_info, &vertical->scale_info );
8858}
8859
8860// The horizontal kernels for this channel count and coefficient width.
8861static void stbir__select_horizontal_kernels( stbir__info * info, stbir__sampler const * horizontal, int effective_channels )
8862{
8863 int widest = horizontal->extent_info.widest;
8864
8865 // start with defaulting to the n_coeffs functions (specialized on channels and remnant leftover)
8866 info->horizontal_gather_channels = stbir__horizontal_gather_n_coeffs_funcs[ effective_channels ][ widest & 3 ];
8867
8868 // but if the number of coeffs <= 17, use another set of special cases. This is by far the common case
8869 if ( widest <= 17 )
8870 info->horizontal_gather_channels = stbir__horizontal_gather_channels_funcs[ effective_channels ][ widest - 1 ];
8871
8872 // integer downsamples (polyphase period 1) use one coefficient set for the whole interior, so the kernel can keep the
8873 // coefficients in registers for the whole run - where there is one for this channel count and width (null otherwise)
8874 info->horizontal_gather_period1 = 0;
8875 if ( ( horizontal->shared_period == 1 ) && ( widest <= 17 ) )
8876 info->horizontal_gather_period1 = stbir__horizontal_gather_period1_funcs[ effective_channels ][ widest ];
8877}
8878
8879// The per split buffers are sized for one column slice (the slices run one after another and reuse them).
8880typedef struct stbir__slice_buffer_sizes
8881{
8882 int num_column_slices; // a multiple of the column groups, so they get equal shares
8883 int slice_columns; // the widest slice
8884 int slice_in_span; // the most input pixels a slice decodes
8885 size_t decode_buffer_size;
8886 size_t ring_buffer_length_bytes; // one ring entry
8887 size_t ring_buffer_size;
8888 size_t vertical_buffer_size;
8889} stbir__slice_buffer_sizes;
8890
8891static void stbir__size_slice_buffers( stbir__slice_buffer_sizes * sizes, stbir__sampler const * horizontal, int conservative_in_span, int l2_slices, int split_cols, int effective_channels, int vertical_first, int ring_buffer_num_entries )
8892{
8893 int output_columns = horizontal->scale_info.output_sub_size;
8894 int max_slices = stbir__slice_max_count( output_columns );
8895
8896 // the slice count made a multiple of the column groups (a slice is at least a step wide, so some groups may get none)
8897 sizes->num_column_slices = ( ( l2_slices + split_cols - 1 ) / split_cols ) * split_cols;
8898 if ( sizes->num_column_slices > max_slices )
8899 sizes->num_column_slices = max_slices;
8900 sizes->slice_columns = stbir__slice_columns_for( output_columns, sizes->num_column_slices );
8901
8902 // the input pixels a slice decodes: its columns' geometric span, plus the filter width, plus the room
8903 // stbir__pack_coefficients can move a contributor back by and the kernel can read past it (a coefficient
8904 // width each). the exact ranges come from the packed contributors in the second pass, which asserts
8905 // they fit in this. a single slice is exactly the conservative range.
8906 sizes->slice_in_span = stbir__slice_in_span_for( horizontal, sizes->slice_columns, conservative_in_span );
8907
8908 // sometimes read one float off in some of the unrolled loops (with a weight of zero coeff, so it doesn't have an effect)
8909 // we use a few extra floats instead of just 1, so that input callback buffer can overlap with the decode buffer without
8910 // the conversion routines overwriting the callback input data.
8911 sizes->decode_buffer_size = (size_t)sizes->slice_in_span * effective_channels * sizeof(float) + sizeof(float)*STBIR_INPUT_CALLBACK_PADDING; // extra floats for input callback stagger
8912
8913#if defined( STBIR__SEPARATE_ALLOCATIONS ) && defined(STBIR_SIMD8)
8914 if ( effective_channels == 3 )
8915 sizes->decode_buffer_size += sizeof(float); // avx in 3 channel mode needs one float at the start of the buffer (only with separate allocations)
8916#endif
8917
8918 sizes->ring_buffer_length_bytes = (size_t)sizes->slice_columns * (size_t)effective_channels * sizeof(float) + sizeof(float)*STBIR_INPUT_CALLBACK_PADDING; // extra floats for padding
8919
8920 // if we do vertical first, the ring buffer holds a whole decoded line
8921 if ( vertical_first )
8922 sizes->ring_buffer_length_bytes = ( sizes->decode_buffer_size + 15 ) & ~15;
8923
8924 // keep the ring buffer's entries off a page multiple apart: the vertical gather reads taps rows at the same column,
8925 // and at a 4K multiple stride they all fall in the same cache sets
8926 if ( ( sizes->ring_buffer_length_bytes & 4095 ) == 0 )
8927 sizes->ring_buffer_length_bytes += 64*3;
8928
8929 sizes->ring_buffer_size = (size_t)ring_buffer_num_entries * sizes->ring_buffer_length_bytes;
8930
8931 // The vertical buffer holds one horizontally resampled row: gathering, the output row (the vertical gather's result
8932 // horizontal first, the horizontal's result vertical first); scattering, the input row about to be scattered
8933 // (horizontal first) or the output row at eviction (vertical first). The accumulation itself is in the ring.
8934 sizes->vertical_buffer_size = (size_t)sizes->slice_columns * (size_t)effective_channels * sizeof(float) + sizeof(float); // extra float for padding
8935}
8936
8937static stbir__info * stbir__alloc_internal_mem_and_build_samplers( stbir__sampler * horizontal, stbir__sampler * vertical, stbir__contributors * conservative, stbir_pixel_layout input_pixel_layout_public, stbir_pixel_layout output_pixel_layout_public, int splits, int new_x, int new_y, int fast_alpha, int input_pixel_bytes, void * user_data STBIR_ONLY_PROFILE_BUILD_GET_INFO )
8938{
8939 static char stbir_channel_count_index[8]={ 9,0,1,2, 3,9,9,4 };
8940
8941 stbir__info * info = 0;
8942 void * alloced = 0;
8943 size_t alloced_total = 0;
8944 int vertical_first;
8945 int alloc_ring_buffer_num_entries;
8946 int alpha_weighting_type;
8947 int conservative_split_output_size;
8948 int conservative_in_span, l2_slices, split_cols, split_rows;
8949 int copy_horizontal;
8950 stbir__slice_buffer_sizes sizes;
8951 stbir_internal_pixel_layout input_pixel_layout = stbir__pixel_layout_convert_public_to_internal[ input_pixel_layout_public ];
8952 stbir_internal_pixel_layout output_pixel_layout = stbir__pixel_layout_convert_public_to_internal[ output_pixel_layout_public ];
8953 int channels = stbir__pixel_channels[ input_pixel_layout ];
8954 int effective_channels = channels;
8955
8956 // channel in and out count must match currently
8957 if ( channels != stbir__pixel_channels[ output_pixel_layout ] )
8958 return 0;
8959
8960 alpha_weighting_type = stbir__alpha_weighting_type( horizontal, vertical, input_pixel_layout, output_pixel_layout, fast_alpha, &effective_channels );
8961
8962 // do the horizontal and vertical store their polyphase coefficient sets once? (sizes the tables, so before anything
8963 // is allocated). the vertical only when gathering (the scatter tables are per input scanline). the vertical kernels
8964 // read exactly n0..n1 of a set, so its epilogue needs no clip zone.
8965 stbir__plan_polyphase_sharing( horizontal, 1, user_data );
8966 if ( vertical->is_gather )
8967 stbir__plan_polyphase_sharing( vertical, 0, user_data );
8968
8969 vertical_first = stbir__should_do_vertical_first( (int)stbir_channel_count_index[ effective_channels ],
8970 horizontal->filter_pixel_width, horizontal->scale_info.input_full_size, horizontal->scale_info.output_sub_size, horizontal->edge, horizontal->shared_period == 1,
8971 vertical->filter_pixel_width, vertical->scale_info.input_full_size, vertical->scale_info.output_sub_size, vertical->edge );
8972
8973 // ring rows: gathering, an output's taps rows (plus one for float rounding); scattering, the output rows an input
8974 // row touches - 2 x the filter's support in output pixels, plus one, plus one for rounding
8975 if ( vertical->is_gather )
8976 alloc_ring_buffer_num_entries = vertical->filter_pixel_width + 1;
8977 else
8978 alloc_ring_buffer_num_entries = stbir__ceil_to_int( (stbir_real)2.0 * stbir__filter_radius_in_output_pixels( vertical->filter_radius, vertical->filter_support_new, vertical->filter_support_old, &vertical->scale_info, user_data ) ) + 2;
8979
8980 // how many vertical slices the single threaded resize would run (the setting, or the L2 fit), then the split grid -
8981 // which can lower the split count
8982 conservative_in_span = conservative->n1 - conservative->n0 + 1;
8983 l2_slices = stbir__column_slices;
8984 if ( l2_slices < 1 )
8985 l2_slices = stbir__choose_column_slices( horizontal, conservative_in_span, effective_channels, vertical_first, alloc_ring_buffer_num_entries );
8986 stbir__choose_split_grid( &splits, &split_cols, &split_rows, stbir__split_mode, horizontal->scale_info.output_sub_size, vertical->scale_info.output_sub_size, conservative_in_span, input_pixel_bytes, vertical->filter_pixel_width, (float) vertical->scale_info.inv_scale, l2_slices, effective_channels );
8987 conservative_split_output_size = stbir__get_max_split( split_rows, vertical->scale_info.output_sub_size );
8988
8989 // we never need more ring buffer entries than the scanlines we're outputting when in scatter mode
8990 if ( ( !vertical->is_gather ) && ( alloc_ring_buffer_num_entries > conservative_split_output_size ) )
8991 alloc_ring_buffer_num_entries = conservative_split_output_size;
8992
8993 stbir__size_slice_buffers( &sizes, horizontal, conservative_in_span, l2_slices, split_cols, effective_channels, vertical_first, alloc_ring_buffer_num_entries );
8994
8995 // are the two filters identical? then the vertical shares the horizontal's tables instead of generating its own
8996 copy_horizontal = stbir__samplers_match( horizontal, vertical );
8997
8998 // we make two passes through this loop, 1st to add everything up, 2nd to allocate and init
8999 for(;;)
9000 {
9001 int i;
9002 void * advance_mem = alloced;
9003
9004#ifdef STBIR__SEPARATE_ALLOCATIONS
9005 // a failed allocation frees whatever got allocated: the pointers the cleanup frees are cleared as their holders are
9006 // allocated, so every one is an allocation or null (if info itself failed, only the placeholder block exists)
9007 #define STBIR__NEXT_PTR( ptr, size, ntype ) if ( alloced ) { void * p = STBIR_MALLOC( size, user_data); if ( p == 0 ) { if ( info ) stbir__free_internal_mem( info ); else STBIR_FREE( alloced, user_data ); return 0; } (ptr) = (ntype*)p; }
9008#else
9009 #define STBIR__NEXT_PTR( ptr, size, ntype ) advance_mem = (void*) ( ( ((size_t)advance_mem) + 15 ) & ~15 ); if ( alloced ) ptr = (ntype*)advance_mem; advance_mem = (char*)(((size_t)advance_mem) + (size));
9010#endif
9011
9012 STBIR__NEXT_PTR( info, sizeof( stbir__info ), stbir__info );
9013 if ( info )
9014 {
9015 info->alloced_mem = alloced;
9016 info->user_data = user_data;
9017 info->splits = splits;
9018 info->alloc_ring_buffer_num_entries = (int)alloc_ring_buffer_num_entries;
9019 info->effective_channels = effective_channels;
9020 info->split_info = 0;
9021 info->column_slices = 0;
9022 info->horizontal.contributors = 0;
9023 info->horizontal.coefficients = 0;
9024 info->vertical.contributors = 0;
9025 info->vertical.coefficients = 0;
9026 }
9027
9028 STBIR__NEXT_PTR( info->split_info, sizeof( stbir__per_split_info ) * splits, stbir__per_split_info );
9029 #ifdef STBIR__SEPARATE_ALLOCATIONS
9030 if ( info )
9031 {
9032 for( i = 0 ; i < splits ; i++ )
9033 {
9034 info->split_info[i].ring_buffers = 0;
9035 info->split_info[i].decode_buffer_alloc = 0;
9036 info->split_info[i].vertical_buffer_alloc = 0;
9037 }
9038 }
9039 #endif
9040
9041 if ( info )
9042 {
9043 // initialize info fields
9044 info->alloced_total = alloced_total;
9045 info->channels = channels;
9046 info->offset_x = new_x;
9047 info->offset_y = new_y;
9048 info->ring_buffer_num_entries = 0;
9049 info->ring_buffer_length_bytes = sizes.ring_buffer_length_bytes;
9050 info->vertical_first = vertical_first;
9051
9052 info->input_pixel_layout_internal = input_pixel_layout;
9053 info->output_pixel_layout_internal = output_pixel_layout;
9054
9055 stbir__set_alpha_functions( info, alpha_weighting_type, input_pixel_layout, output_pixel_layout, (float) horizontal->scale_info.scale );
9056 }
9057
9058 // get all the per-split buffers (each with a page of slack, see stbir__4K_alias_skew_buffers)
9059 for( i = 0 ; i < splits ; i++ )
9060 {
9061 STBIR__NEXT_PTR( info->split_info[i].decode_buffer, sizes.decode_buffer_size + STBIR__4K_ALIAS_SKEW_SLACK, float );
9062
9063#ifdef STBIR__SEPARATE_ALLOCATIONS
9064
9065 #ifdef STBIR_SIMD8
9066 if ( ( info ) && ( effective_channels == 3 ) )
9067 ++info->split_info[i].decode_buffer; // avx in 3 channel mode needs one float at the start of the buffer
9068 #endif
9069 if ( info )
9070 info->split_info[i].decode_buffer_alloc = info->split_info[i].decode_buffer;
9071
9072 STBIR__NEXT_PTR( info->split_info[i].ring_buffers, alloc_ring_buffer_num_entries * sizeof(float*), float* );
9073 {
9074 int j;
9075 if ( info )
9076 {
9077 for( j = 0 ; j < alloc_ring_buffer_num_entries ; j++ )
9078 info->split_info[i].ring_buffers[j] = 0;
9079 }
9080 for( j = 0 ; j < alloc_ring_buffer_num_entries ; j++ )
9081 {
9082 STBIR__NEXT_PTR( info->split_info[i].ring_buffers[j], sizes.ring_buffer_length_bytes, float );
9083 #ifdef STBIR_SIMD8
9084 if ( ( info ) && ( effective_channels == 3 ) )
9085 ++info->split_info[i].ring_buffers[j]; // avx in 3 channel mode needs one float at the start of the buffer
9086 #endif
9087 }
9088 }
9089#else
9090 STBIR__NEXT_PTR( info->split_info[i].ring_buffer, sizes.ring_buffer_size + STBIR__4K_ALIAS_SKEW_SLACK, float );
9091#endif
9092 STBIR__NEXT_PTR( info->split_info[i].vertical_buffer, sizes.vertical_buffer_size + STBIR__4K_ALIAS_SKEW_SLACK, float );
9093
9094 if ( info )
9095 {
9096 #ifndef STBIR__SEPARATE_ALLOCATIONS
9097 info->split_info[i].decode_buffer_alloc = info->split_info[i].decode_buffer;
9098 info->split_info[i].ring_buffer_alloc = info->split_info[i].ring_buffer;
9099 #endif
9100 info->split_info[i].vertical_buffer_alloc = info->split_info[i].vertical_buffer;
9101 }
9102 }
9103
9104 // the tables are allocated into info's samplers, and the samplers are built in the locals (copied into info below)
9105 STBIR__NEXT_PTR( info->horizontal.contributors, horizontal->contributors_size, stbir__contributors );
9106 STBIR__NEXT_PTR( info->horizontal.coefficients, horizontal->coefficients_size, float );
9107
9108 if ( !copy_horizontal ) // (sharing: the vertical reads the horizontal's tables, see below)
9109 {
9110 STBIR__NEXT_PTR( info->vertical.contributors, vertical->contributors_size, stbir__contributors );
9111 STBIR__NEXT_PTR( info->vertical.coefficients, vertical->coefficients_size, float );
9112 }
9113
9114 if ( info )
9115 {
9116 horizontal->contributors = info->horizontal.contributors;
9117 horizontal->coefficients = info->horizontal.coefficients;
9118 vertical->contributors = info->vertical.contributors;
9119 vertical->coefficients = info->vertical.coefficients;
9120 }
9121
9122 STBIR__NEXT_PTR( info->column_slices, sizeof( stbir__column_slice ) * sizes.num_column_slices, stbir__column_slice );
9123
9124 if ( info )
9125 {
9126 STBIR_PROFILE_BUILD_START( horizontal );
9127
9128 stbir__calculate_filters( horizontal, user_data STBIR_ONLY_PROFILE_BUILD_SET_INFO );
9129 stbir__select_horizontal_kernels( info, horizontal, effective_channels );
9130
9131 info->conservative_extents = *conservative;
9132
9133 // pack the horizontal coeffs into the column entries the kernels walk (the vertical's aren't packed: its kernels read exactly n0..n1)
9134 if ( horizontal->shared_period )
9135 horizontal->coefficient_width = stbir__pack_shared_coefficients( horizontal, info->conservative_extents.n0, info->conservative_extents.n1 );
9136 else
9137 horizontal->coefficient_width = stbir__pack_coefficients(horizontal->num_contributors, horizontal->contributors, horizontal->coefficients, horizontal->coefficient_width, horizontal->extent_info.widest, info->conservative_extents.n0, info->conservative_extents.n1 );
9138 horizontal->entry_floats = STBIR__COLUMN_ENTRY_FLOATS( horizontal->coefficient_width );
9139
9140 STBIR_MEMCPY( &info->horizontal, horizontal, sizeof( stbir__sampler ) );
9141
9142 // now that the contributors are final, work out exactly what each column slice decodes
9143 info->num_column_slices = sizes.num_column_slices;
9144 stbir__build_column_slices( info, sizes.slice_columns, sizes.slice_in_span );
9145
9146 STBIR_PROFILE_BUILD_END( horizontal );
9147
9148 STBIR_PROFILE_BUILD_START( vertical );
9149 if ( copy_horizontal )
9150 {
9151 // identical samplers: the vertical reads the horizontal's packed column entries (two ints, then the coefficients)
9152 STBIR_MEMCPY( vertical, horizontal, sizeof( stbir__sampler ) );
9153 vertical->coefficients += 2;
9154 vertical->coefficient_width = horizontal->entry_floats;
9155 }
9156 else
9157 stbir__calculate_filters( vertical, user_data STBIR_ONLY_PROFILE_BUILD_SET_INFO );
9158 STBIR_MEMCPY( &info->vertical, vertical, sizeof( stbir__sampler ) );
9159 STBIR_PROFILE_BUILD_END( vertical );
9160
9161 // the horizontal's contributors have been read for the last time (the slice table), so fill in the column entries'
9162 // advance and width (an advance that doesn't fit fails the build)
9163 if ( !stbir__fill_column_entries( &info->horizontal, effective_channels ) )
9164 {
9165 stbir__free_internal_mem( info );
9166 return 0;
9167 }
9168
9169 stbir__setup_split_grid( info, split_cols, split_rows );
9170
9171 // now we know precisely how many entries we need
9172 info->ring_buffer_num_entries = info->vertical.extent_info.widest;
9173
9174 // we never need more ring buffer entries than the scanlines we're outputting
9175 if ( ( !info->vertical.is_gather ) && ( info->ring_buffer_num_entries > conservative_split_output_size ) )
9176 info->ring_buffer_num_entries = conservative_split_output_size;
9177 STBIR_ASSERT( info->ring_buffer_num_entries <= info->alloc_ring_buffer_num_entries );
9178 }
9179 #undef STBIR__NEXT_PTR
9180
9181 // is this the first time through loop?
9182 if ( info == 0 )
9183 {
9184 alloced_total = ( 15 + (size_t)advance_mem );
9185 alloced = STBIR_MALLOC( alloced_total, user_data );
9186 if ( alloced == 0 )
9187 return 0;
9188 }
9189 else
9190 return info; // success
9191 }
9192}
9193
9194// 4K aliasing avoidance: each internal buffer has a page of slack and is placed half a page from the buffer
9195// that streams into it - input -> decode -> ring -> vertical buffer (horizontal first),
9196// input -> ring -> decode -> vertical buffer (vertical first).
9197static size_t stbir__4K_alias_place( void * alloc, size_t want )
9198{
9199 size_t have = ( (size_t)alloc ) & 4095;
9200 return ( ( want - have ) & 4095 ) & ~(size_t)15; // stays 16 byte aligned, within the slack
9201}
9202
9203static void stbir__4K_alias_skew_buffers( stbir__info const * info, stbir__per_split_info * split_info )
9204{
9205 size_t first = ( ( (size_t)info->input_data ) + 2048 ) & 4095; // the first internal buffer: half a page from the input row
9206 size_t second = ( first + 2048 ) & 4095; // and the next one, back at the input's offset
9207 size_t vertical_dest;
9208
9209 if ( info->vertical_first )
9210 {
9211 // decode: input -> ring. vertical: ring -> decode_buffer. horizontal: decode_buffer -> vertical_buffer. encode: vertical_buffer -> output
9212 #ifndef STBIR__SEPARATE_ALLOCATIONS
9213 split_info->ring_buffer = (float*) ( (char*)split_info->ring_buffer_alloc + stbir__4K_alias_place( split_info->ring_buffer_alloc, first ) );
9214 #endif
9215 vertical_dest = ( second + (size_t)stbir__4K_vertical_dest_shift ) & 4095;
9216 split_info->decode_buffer = (float*) ( (char*)split_info->decode_buffer_alloc + stbir__4K_alias_place( split_info->decode_buffer_alloc, vertical_dest ) );
9217 split_info->vertical_buffer = (float*) ( (char*)split_info->vertical_buffer_alloc + stbir__4K_alias_place( split_info->vertical_buffer_alloc, ( vertical_dest + 2048 ) & 4095 ) );
9218 }
9219 else
9220 {
9221 // decode: input -> decode_buffer. horizontal: decode_buffer -> ring. vertical: ring -> vertical_buffer. encode: vertical_buffer -> output
9222 split_info->decode_buffer = (float*) ( (char*)split_info->decode_buffer_alloc + stbir__4K_alias_place( split_info->decode_buffer_alloc, first ) );
9223 #ifndef STBIR__SEPARATE_ALLOCATIONS
9224 split_info->ring_buffer = (float*) ( (char*)split_info->ring_buffer_alloc + stbir__4K_alias_place( split_info->ring_buffer_alloc, second ) );
9225 #endif
9226 vertical_dest = ( first + (size_t)stbir__4K_vertical_dest_shift ) & 4095;
9227 split_info->vertical_buffer = (float*) ( (char*)split_info->vertical_buffer_alloc + stbir__4K_alias_place( split_info->vertical_buffer_alloc, vertical_dest ) );
9228 }
9229}
9230
9231static int stbir__perform_resize( stbir__info const * info, int split_start, int split_count )
9232{
9233 stbir__per_split_info * split_info = info->split_info + split_start;
9234 int b, i, can_merge;
9235
9236 for( i = 0 ; i < split_count ; i++ )
9237 stbir__4K_alias_skew_buffers( info, split_info + i );
9238
9239 STBIR_PROFILE_CLEAR_EXTRAS();
9240
9241 STBIR_PROFILE_FIRST_START( looping );
9242 // run the whole vertical loop once per column slice, so each pass only touches a cache sized slice of the buffers.
9243 // consecutive splits over the same slices are adjacent row bands and run as one band (one warm-up per slice) - except
9244 // when a scatter's ring was cut down to a band's rows: an input row can touch more output rows than that across
9245 // two bands, so those run one at a time
9246 can_merge = info->vertical.is_gather || ( info->ring_buffer_num_entries >= info->vertical.extent_info.widest );
9247 i = 0;
9248 while ( i < split_count )
9249 {
9250 int run = 1;
9251 while ( can_merge && ( i + run < split_count ) && ( split_info[i+run].first_column_slice == split_info[i].first_column_slice ) && ( split_info[i+run].num_column_slices == split_info[i].num_column_slices ) )
9252 ++run;
9253 STBIR_PROFILE_NEST_IN_FIRST( split_info + i );
9254 for( b = 0 ; b < split_info[i].num_column_slices ; b++ )
9255 {
9256 stbir__column_slice const * slice = info->column_slices + split_info[i].first_column_slice + b;
9257 if (info->vertical.is_gather)
9258 stbir__vertical_gather_loop( info, split_info + i, run, slice );
9259 else
9260 stbir__vertical_scatter_loop( info, split_info + i, run, slice );
9261 }
9262 i += run;
9263 }
9264 STBIR_PROFILE_END( looping );
9265
9266 return 1;
9267}
9268
9269static void stbir__update_info_from_resize( stbir__info * info, STBIR_RESIZE * resize )
9270{
9271 static stbir__decode_pixels_func * decode_simple[STBIR_TYPE_HALF_FLOAT-STBIR_TYPE_UINT8_SRGB+1]=
9272 {
9273 /* 1ch-4ch */ stbir__decode_uint8_srgb, stbir__decode_uint8_srgb, 0, stbir__decode_float_linear, stbir__decode_half_float_linear,
9274 };
9275
9276 static stbir__decode_pixels_func * decode_alphas[STBIRI_AR-STBIRI_RGBA+1][STBIR_TYPE_HALF_FLOAT-STBIR_TYPE_UINT8_SRGB+1]=
9277 {
9278 { /* RGBA */ stbir__decode_uint8_srgb4_linearalpha, stbir__decode_uint8_srgb, 0, stbir__decode_float_linear, stbir__decode_half_float_linear },
9279 { /* BGRA */ stbir__decode_uint8_srgb4_linearalpha_BGRA, stbir__decode_uint8_srgb_BGRA, 0, stbir__decode_float_linear_BGRA, stbir__decode_half_float_linear_BGRA },
9280 { /* ARGB */ stbir__decode_uint8_srgb4_linearalpha_ARGB, stbir__decode_uint8_srgb_ARGB, 0, stbir__decode_float_linear_ARGB, stbir__decode_half_float_linear_ARGB },
9281 { /* ABGR */ stbir__decode_uint8_srgb4_linearalpha_ABGR, stbir__decode_uint8_srgb_ABGR, 0, stbir__decode_float_linear_ABGR, stbir__decode_half_float_linear_ABGR },
9282 { /* RA */ stbir__decode_uint8_srgb2_linearalpha, stbir__decode_uint8_srgb, 0, stbir__decode_float_linear, stbir__decode_half_float_linear },
9283 { /* AR */ stbir__decode_uint8_srgb2_linearalpha_AR, stbir__decode_uint8_srgb_AR, 0, stbir__decode_float_linear_AR, stbir__decode_half_float_linear_AR },
9284 };
9285
9286 static stbir__decode_pixels_func * decode_simple_scaled_or_not[2][2]=
9287 {
9288 { stbir__decode_uint8_linear_scaled, stbir__decode_uint8_linear }, { stbir__decode_uint16_linear_scaled, stbir__decode_uint16_linear },
9289 };
9290
9291 static stbir__decode_pixels_func * decode_alphas_scaled_or_not[STBIRI_AR-STBIRI_RGBA+1][2][2]=
9292 {
9293 { /* RGBA */ { stbir__decode_uint8_linear_scaled, stbir__decode_uint8_linear }, { stbir__decode_uint16_linear_scaled, stbir__decode_uint16_linear } },
9294 { /* BGRA */ { stbir__decode_uint8_linear_scaled_BGRA, stbir__decode_uint8_linear_BGRA }, { stbir__decode_uint16_linear_scaled_BGRA, stbir__decode_uint16_linear_BGRA } },
9295 { /* ARGB */ { stbir__decode_uint8_linear_scaled_ARGB, stbir__decode_uint8_linear_ARGB }, { stbir__decode_uint16_linear_scaled_ARGB, stbir__decode_uint16_linear_ARGB } },
9296 { /* ABGR */ { stbir__decode_uint8_linear_scaled_ABGR, stbir__decode_uint8_linear_ABGR }, { stbir__decode_uint16_linear_scaled_ABGR, stbir__decode_uint16_linear_ABGR } },
9297 { /* RA */ { stbir__decode_uint8_linear_scaled, stbir__decode_uint8_linear }, { stbir__decode_uint16_linear_scaled, stbir__decode_uint16_linear } },
9298 { /* AR */ { stbir__decode_uint8_linear_scaled_AR, stbir__decode_uint8_linear_AR }, { stbir__decode_uint16_linear_scaled_AR, stbir__decode_uint16_linear_AR } }
9299 };
9300
9301 static stbir__encode_pixels_func * encode_simple[STBIR_TYPE_HALF_FLOAT-STBIR_TYPE_UINT8_SRGB+1]=
9302 {
9303 /* 1ch-4ch */ stbir__encode_uint8_srgb, stbir__encode_uint8_srgb, 0, stbir__encode_float_linear, stbir__encode_half_float_linear,
9304 };
9305
9306 static stbir__encode_pixels_func * encode_alphas[STBIRI_AR-STBIRI_RGBA+1][STBIR_TYPE_HALF_FLOAT-STBIR_TYPE_UINT8_SRGB+1]=
9307 {
9308 { /* RGBA */ stbir__encode_uint8_srgb4_linearalpha, stbir__encode_uint8_srgb, 0, stbir__encode_float_linear, stbir__encode_half_float_linear },
9309 { /* BGRA */ stbir__encode_uint8_srgb4_linearalpha_BGRA, stbir__encode_uint8_srgb_BGRA, 0, stbir__encode_float_linear_BGRA, stbir__encode_half_float_linear_BGRA },
9310 { /* ARGB */ stbir__encode_uint8_srgb4_linearalpha_ARGB, stbir__encode_uint8_srgb_ARGB, 0, stbir__encode_float_linear_ARGB, stbir__encode_half_float_linear_ARGB },
9311 { /* ABGR */ stbir__encode_uint8_srgb4_linearalpha_ABGR, stbir__encode_uint8_srgb_ABGR, 0, stbir__encode_float_linear_ABGR, stbir__encode_half_float_linear_ABGR },
9312 { /* RA */ stbir__encode_uint8_srgb2_linearalpha, stbir__encode_uint8_srgb, 0, stbir__encode_float_linear, stbir__encode_half_float_linear },
9313 { /* AR */ stbir__encode_uint8_srgb2_linearalpha_AR, stbir__encode_uint8_srgb_AR, 0, stbir__encode_float_linear_AR, stbir__encode_half_float_linear_AR }
9314 };
9315
9316 static stbir__encode_pixels_func * encode_simple_scaled_or_not[2][2]=
9317 {
9318 { stbir__encode_uint8_linear_scaled, stbir__encode_uint8_linear }, { stbir__encode_uint16_linear_scaled, stbir__encode_uint16_linear },
9319 };
9320
9321 static stbir__encode_pixels_func * encode_alphas_scaled_or_not[STBIRI_AR-STBIRI_RGBA+1][2][2]=
9322 {
9323 { /* RGBA */ { stbir__encode_uint8_linear_scaled, stbir__encode_uint8_linear }, { stbir__encode_uint16_linear_scaled, stbir__encode_uint16_linear } },
9324 { /* BGRA */ { stbir__encode_uint8_linear_scaled_BGRA, stbir__encode_uint8_linear_BGRA }, { stbir__encode_uint16_linear_scaled_BGRA, stbir__encode_uint16_linear_BGRA } },
9325 { /* ARGB */ { stbir__encode_uint8_linear_scaled_ARGB, stbir__encode_uint8_linear_ARGB }, { stbir__encode_uint16_linear_scaled_ARGB, stbir__encode_uint16_linear_ARGB } },
9326 { /* ABGR */ { stbir__encode_uint8_linear_scaled_ABGR, stbir__encode_uint8_linear_ABGR }, { stbir__encode_uint16_linear_scaled_ABGR, stbir__encode_uint16_linear_ABGR } },
9327 { /* RA */ { stbir__encode_uint8_linear_scaled, stbir__encode_uint8_linear }, { stbir__encode_uint16_linear_scaled, stbir__encode_uint16_linear } },
9328 { /* AR */ { stbir__encode_uint8_linear_scaled_AR, stbir__encode_uint8_linear_AR }, { stbir__encode_uint16_linear_scaled_AR, stbir__encode_uint16_linear_AR } }
9329 };
9330
9331 stbir__decode_pixels_func * decode_pixels = 0;
9332 stbir__encode_pixels_func * encode_pixels = 0;
9333 stbir_datatype input_type, output_type;
9334 int non_scaled;
9335
9336 input_type = resize->input_data_type;
9337 output_type = resize->output_data_type;
9338 info->input_data = resize->input_pixels;
9339 info->input_stride_bytes = resize->input_stride_in_bytes;
9340 info->output_stride_bytes = resize->output_stride_in_bytes;
9341
9342 // if we're completely point sampling, then we can turn off SRGB
9343 if ( ( info->horizontal.filter_enum == STBIR_FILTER_POINT_SAMPLE ) && ( info->vertical.filter_enum == STBIR_FILTER_POINT_SAMPLE ) )
9344 {
9345 if ( ( ( input_type == STBIR_TYPE_UINT8_SRGB ) || ( input_type == STBIR_TYPE_UINT8_SRGB_ALPHA ) ) &&
9346 ( ( output_type == STBIR_TYPE_UINT8_SRGB ) || ( output_type == STBIR_TYPE_UINT8_SRGB_ALPHA ) ) )
9347 {
9348 input_type = STBIR_TYPE_UINT8;
9349 output_type = STBIR_TYPE_UINT8;
9350 }
9351 }
9352
9353 // recalc the output and input strides (zero means packed - the whole image's width, the sub rect is a region of it)
9354 if ( info->input_stride_bytes == 0 )
9355 info->input_stride_bytes = (stbir_int64)info->channels * info->horizontal.scale_info.input_full_size * stbir__type_size[input_type];
9356
9357 if ( info->output_stride_bytes == 0 )
9358 info->output_stride_bytes = (stbir_int64)info->channels * resize->output_w * stbir__type_size[output_type];
9359
9360 // calc offset
9361 info->output_data = ( (char*) resize->output_pixels ) + (ptrdiff_t)( info->offset_y * info->output_stride_bytes ) + ( (size_t) info->offset_x * (size_t) info->channels * (size_t) stbir__type_size[output_type] );
9362
9363 info->in_pixels_cb = resize->input_cb;
9364 info->user_data = resize->user_data;
9365 info->out_pixels_cb = resize->output_cb;
9366
9367 // check if we can run unscaled - 0-255.0/0-65535.0 instead of 0-1.0 (which is a tiny bit faster when doing linear 8->8 or 16->16)
9368 // don't short circuit when alpha weighting (get everything to 0-1.0 as usual)
9369 non_scaled = 0;
9370 if ( ( !info->alpha_weight ) && ( !info->alpha_unweight ) )
9371 if ( ( ( input_type == STBIR_TYPE_UINT8 ) && ( output_type == STBIR_TYPE_UINT8 ) ) || ( ( input_type == STBIR_TYPE_UINT16 ) && ( output_type == STBIR_TYPE_UINT16 ) ) )
9372 non_scaled = 1;
9373
9374 // setup the input format converters
9375 if ( ( input_type == STBIR_TYPE_UINT8 ) || ( input_type == STBIR_TYPE_UINT16 ) )
9376 {
9377 if ( info->input_pixel_layout_internal <= STBIRI_4CHANNEL )
9378 decode_pixels = decode_simple_scaled_or_not[ input_type == STBIR_TYPE_UINT16 ][ non_scaled ];
9379 else
9380 decode_pixels = decode_alphas_scaled_or_not[ ( info->input_pixel_layout_internal - STBIRI_RGBA ) % ( STBIRI_AR-STBIRI_RGBA+1 ) ][ input_type == STBIR_TYPE_UINT16 ][ non_scaled ];
9381 }
9382 else
9383 {
9384 if ( info->input_pixel_layout_internal <= STBIRI_4CHANNEL )
9385 decode_pixels = decode_simple[ input_type - STBIR_TYPE_UINT8_SRGB ];
9386 else
9387 decode_pixels = decode_alphas[ ( info->input_pixel_layout_internal - STBIRI_RGBA ) % ( STBIRI_AR-STBIRI_RGBA+1 ) ][ input_type - STBIR_TYPE_UINT8_SRGB ];
9388 }
9389
9390 // setup the output format converters
9391 if ( ( output_type == STBIR_TYPE_UINT8 ) || ( output_type == STBIR_TYPE_UINT16 ) )
9392 {
9393 if ( info->output_pixel_layout_internal <= STBIRI_4CHANNEL )
9394 encode_pixels = encode_simple_scaled_or_not[ output_type == STBIR_TYPE_UINT16 ][ non_scaled ];
9395 else
9396 encode_pixels = encode_alphas_scaled_or_not[ ( info->output_pixel_layout_internal - STBIRI_RGBA ) % ( STBIRI_AR-STBIRI_RGBA+1 ) ][ output_type == STBIR_TYPE_UINT16 ][ non_scaled ];
9397 }
9398 else
9399 {
9400 if ( info->output_pixel_layout_internal <= STBIRI_4CHANNEL )
9401 encode_pixels = encode_simple[ output_type - STBIR_TYPE_UINT8_SRGB ];
9402 else
9403 encode_pixels = encode_alphas[ ( info->output_pixel_layout_internal - STBIRI_RGBA ) % ( STBIRI_AR-STBIRI_RGBA+1 ) ][ output_type - STBIR_TYPE_UINT8_SRGB ];
9404 }
9405
9406 info->input_type = input_type;
9407 info->output_type = output_type;
9408 info->decode_pixels = decode_pixels;
9409 info->encode_pixels = encode_pixels;
9410
9411 // the input callback's temp buffer sits inside the decode buffer, pushed forward so the decoder reads ahead of
9412 // its writes - except plain float, which is a no-op decode when the temp is exactly the decode target
9413 info->input_callback_offset_bytes = ( decode_pixels == stbir__decode_float_linear ) ? 0 : ( sizeof(float) * STBIR_INPUT_CALLBACK_PADDING );
9414}
9415
9416static void stbir__clip( int * outx, int * outsubw, int outw, stbir_real * u0, stbir_real * u1 )
9417{
9418 stbir_real per, adj;
9419 int over;
9420
9421 // do left/top edge
9422 if ( *outx < 0 )
9423 {
9424 per = ( (stbir_real)*outx ) / ( (stbir_real)*outsubw ); // is negative
9425 adj = per * ( *u1 - *u0 );
9426 *u0 -= adj; // increases u0
9427 *outsubw += *outx; // and drop the clipped off pixels from the output count
9428 *outx = 0;
9429 }
9430
9431 // do right/bot edge
9432 over = outw - ( *outx + *outsubw );
9433 if ( over < 0 )
9434 {
9435 per = ( (stbir_real)over ) / ( (stbir_real)*outsubw ); // is negative
9436 adj = per * ( *u1 - *u0 );
9437 *u1 += adj; // decrease u1
9438 *outsubw = outw - *outx;
9439 }
9440}
9441
9442static stbir_uint64 stbir__gcd( stbir_uint64 a, stbir_uint64 b )
9443{
9444 while ( b )
9445 {
9446 stbir_uint64 remainder = a % b;
9447 a = b;
9448 b = remainder;
9449 }
9450 return a;
9451}
9452
9453// The simplest fraction within tolerance of f (f>0): its continued fraction's first convergent that is, in 31 bits (or 0).
9454static int stbir__real_to_fraction( stbir_real f, stbir_real tolerance, stbir_uint32 * out_numer, stbir_uint32 * out_denom )
9455{
9456 stbir_real rem = f;
9457 stbir_uint64 numer = 1, denom = 0, numer_last = 0, denom_last = 1; // the convergents numer / denom
9458 int i;
9459
9460 if ( f <= (stbir_real) 0.0 )
9461 return 0;
9462 for( i = 0 ; i < (int) sizeof( stbir_real ) * 8 ; i++ ) // (a term is at least a bit of f)
9463 {
9464 stbir_uint64 term = (stbir_uint64) rem; // the floor (rem is positive)
9465 stbir_uint64 next;
9466 stbir_real frac, err;
9467
9468 if ( ( term > 0x7fffffff ) || ( ( term * numer + numer_last ) > 0x7fffffff ) || ( ( term * denom + denom_last ) > 0x7fffffff ) )
9469 return 0;
9470 next = term * numer + numer_last; numer_last = numer; numer = next;
9471 next = term * denom + denom_last; denom_last = denom; denom = next;
9472
9473 err = (stbir_real) numer / (stbir_real) denom - f;
9474 if ( err < (stbir_real) 0.0 )
9475 err = -err;
9476 if ( err <= tolerance )
9477 {
9478 *out_numer = (stbir_uint32) numer;
9479 *out_denom = (stbir_uint32) denom;
9480 return 1;
9481 }
9482
9483 frac = rem - (stbir_real) term;
9484 if ( frac <= (stbir_real) 0.0 )
9485 return 0; // (the fraction is exactly f, and still not within tolerance - can't happen for a tolerance >= 0)
9486 rem = (stbir_real) 1.0 / frac;
9487 }
9488 return 0;
9489}
9490
9491// try to find approximation that stays under the drift value: the far edge of the output moves by less than
9492// 1 / STBIR__POLYPHASE_DRIFT_DIV of an output pixel
9493#define STBIR__POLYPHASE_DRIFT_DIV 8192
9494static void stbir__approximate_polyphase( stbir_uint64 numer, stbir_uint64 denom, stbir_uint64 outputs, stbir_uint64 * out_numer, stbir_uint64 * out_denom, int * out_drift_in_pixels )
9495{
9496 stbir_uint64 top = numer, bot = denom;
9497 stbir_uint64 p = 1, q = 0, p_last = 0, q_last = 1;
9498
9499 while ( bot )
9500 {
9501 stbir_uint64 est = top / bot;
9502 stbir_uint64 remainder = top % bot;
9503 stbir_uint64 next, pd, qn, diff;
9504
9505 top = bot;
9506 bot = remainder;
9507
9508 // the next convergent p / q
9509 next = est * p + p_last;
9510 p_last = p;
9511 p = next;
9512
9513 next = est * q + q_last;
9514 q_last = q;
9515 q = next;
9516
9517 // no simpler fraction is close enough (the convergents only grow from here)
9518 if ( p >= outputs )
9519 return;
9520
9521 // its drift at the far edge, in output pixels: outputs x | p D - q N | / ( q N )
9522 pd = p * denom;
9523 qn = q * numer;
9524 diff = ( pd > qn ) ? pd - qn : qn - pd;
9525 if ( diff <= qn / ( outputs * STBIR__POLYPHASE_DRIFT_DIV ) )
9526 {
9527 // that drift at the far edge in input pixels (times denom / numer), rounded up - for the conservative range
9528 stbir_real drift_in = (stbir_real)outputs * (stbir_real)diff * (stbir_real)denom / ( (stbir_real)qn * (stbir_real)numer );
9529 *out_numer = p;
9530 *out_denom = q;
9531 *out_drift_in_pixels = (int)drift_in + 1;
9532 return;
9533 }
9534 }
9535}
9536
9537// The scale as a fraction: output_sub / ( input_full x ( s1 - s0 ) ), with s1 - s0 = ( n1 - n0 ) / d - exact if possible.
9538// Returns 0 when there is no fraction that fits in int.
9539static int stbir__scale_to_rational( int output_sub_range, int input_full_range, stbir_real input_s, int s_n0, int s_n1, int s_d, stbir_uint32 * out_numer, stbir_uint32 * out_denom, int * out_approximated_drift )
9540{
9541 stbir_uint64 numer, denom, g;
9542 stbir_real scale, err;
9543
9544 *out_numer = 0; // (not a fraction: the fields are read before the flag in places, so never leave them unset)
9545 *out_denom = 0;
9546 *out_approximated_drift = 0;
9547
9548 if ( ( s_d > 0 ) && ( s_n1 > s_n0 ) )
9549 {
9550 numer = (stbir_uint64)output_sub_range * (stbir_uint64)s_d;
9551 denom = (stbir_uint64)input_full_range * (stbir_uint64)( s_n1 - s_n0 );
9552 g = stbir__gcd( numer, denom );
9553 numer /= g; denom /= g;
9554 if ( ( numer <= 0x7fffffff ) && ( denom <= 0x7fffffff ) )
9555 {
9556 if ( numer >= (stbir_uint64)output_sub_range )
9557 stbir__approximate_polyphase( numer, denom, (stbir_uint64)output_sub_range, &numer, &denom, out_approximated_drift );
9558 *out_numer = (stbir_uint32) numer;
9559 *out_denom = (stbir_uint32) denom;
9560 return 1;
9561 }
9562 }
9563
9564 scale = (stbir_real)output_sub_range / ( (stbir_real)input_full_range * input_s );
9565 if ( !stbir__real_to_fraction( scale, scale / ( (stbir_real)output_sub_range * (stbir_real)STBIR__POLYPHASE_DRIFT_DIV ), out_numer, out_denom ) )
9566 return 0;
9567 err = (stbir_real)*out_numer / (stbir_real)*out_denom - scale;
9568 if ( err < (stbir_real) 0.0 )
9569 err = -err;
9570 if ( err > (stbir_real) 0.0 )
9571 *out_approximated_drift = (int)( (stbir_real)output_sub_range * err / ( scale * scale ) ) + 1; // the far edge's drift: output pixels x ( err / scale ), in input pixels ( / scale )
9572 return 1;
9573}
9574
9575// clip the input area to the image (and adjust the output area), the counterpart of stbir__clip: the output pixels whose
9576// centers land outside the image are dropped from the ends, so the ones left sample exactly where they did
9577static void stbir__clip_input( int * outx, int * outsubw, stbir_real * u0, stbir_real * u1 )
9578{
9579 stbir_real per_pixel = ( *u1 - *u0 ) / ( (stbir_real)*outsubw ); // input area per output pixel
9580 int drop;
9581
9582 // do left/top edge
9583 if ( *u0 < 0 )
9584 {
9585 drop = stbir__ceil_to_int( -*u0 / per_pixel - (stbir_real)0.5 ); // pixels centered before the image (one right on the edge stays)
9586 if ( drop > *outsubw )
9587 drop = *outsubw;
9588 if ( drop > 0 )
9589 {
9590 *u0 += (stbir_real)drop * per_pixel; // increases u0
9591 *outx += drop;
9592 *outsubw -= drop;
9593 }
9594 }
9595
9596 // do right/bot edge
9597 if ( *u1 > 1 )
9598 {
9599 drop = stbir__ceil_to_int( ( *u1 - 1 ) / per_pixel - (stbir_real)0.5 ); // pixels centered past the image
9600 if ( drop > *outsubw )
9601 drop = *outsubw;
9602 if ( drop > 0 )
9603 {
9604 *u1 -= (stbir_real)drop * per_pixel; // decreases u1
9605 *outsubw -= drop;
9606 }
9607 }
9608}
9609
9610static int stbir__calculate_region_transform( stbir__scale_info * scale_info, int output_full_range, int * output_offset, int output_sub_range, int input_full_range, stbir_real input_s0, stbir_real input_s1, int s_n0, int s_n1, int s_d )
9611{
9612 stbir_real output_range, input_range, output_s, input_s, input_pixels, ratio, unclipped_input_s;
9613 int unclipped_output_sub_range = output_sub_range;
9614 int unclipped_output_offset = *output_offset;
9615
9616 input_s = input_s1 - input_s0;
9617 unclipped_input_s = input_s;
9618
9619 // null area (a negative size is an empty one)
9620 if ( ( output_full_range <= 0 ) || ( input_full_range <= 0 ) ||
9621 ( output_sub_range <= 0 ) || ( input_s <= (stbir_real)0.0 ) )
9622 return 0;
9623
9624 // are either of the ranges completely out of bounds?
9625 if ( ( *output_offset >= output_full_range ) || ( ( *output_offset + output_sub_range ) <= 0 ) || ( input_s0 >= (stbir_real)1.0 ) || ( input_s1 <= (stbir_real)0.0 ) )
9626 return 0;
9627
9628 output_range = (stbir_real)output_full_range;
9629 input_range = (stbir_real)input_full_range;
9630
9631 output_s = ( (stbir_real)output_sub_range) / output_range;
9632
9633 // figure out the scaling to use
9634 ratio = output_s / input_s;
9635
9636 // save scale before clipping: the output pixels over the input pixels the region spans, and its inverse as the same
9637 // two numbers divided the other way - one rounding each, and since a division is monotonic the two agree about 1
9638 // (a scale that rounds to exactly 1 can't have an inverse over 1, which would put a ratio over 1 to the kernels)
9639 input_pixels = input_range * input_s;
9640 scale_info->scale = (stbir_real)output_sub_range / input_pixels;
9641 scale_info->inv_scale = input_pixels / (stbir_real)output_sub_range;
9642
9643 // clip output area to left/right output edges (and adjust input area)
9644 stbir__clip( output_offset, &output_sub_range, output_full_range, &input_s0, &input_s1 );
9645
9646 // clip input area to the image (and adjust output area)
9647 stbir__clip_input( output_offset, &output_sub_range, &input_s0, &input_s1 );
9648
9649 // recalc input area
9650 input_s = input_s1 - input_s0;
9651
9652 // after clipping do we have zero input area?
9653 if ( ( input_s <= (stbir_real)0.0 ) || ( output_sub_range <= 0 ) )
9654 return 0;
9655
9656 // calculate and store the starting source offsets in output pixel space
9657 scale_info->pixel_shift = input_s0 * ratio * output_range;
9658
9659 scale_info->scale_is_rational = stbir__scale_to_rational( unclipped_output_sub_range, input_full_range, unclipped_input_s, s_n0, s_n1, s_d, &scale_info->scale_numerator, &scale_info->scale_denominator, &scale_info->approximated_drift );
9660
9661 // an exact fraction gives the scale itself - correctly rounded, so an exact ratio's filter widths ( ceil( 2 support / scale ) )
9662 // don't pick up an extra tap from the region math landing an ulp under the integer, and a 1:1 is exactly 1 (the up / down
9663 // decision). an approximated fraction leaves the real scale: it is what the kernels see, and the decision has to agree with it
9664 if ( ( scale_info->scale_is_rational ) && ( !scale_info->approximated_drift ) )
9665 {
9666 scale_info->scale = (stbir_real) scale_info->scale_numerator / (stbir_real) scale_info->scale_denominator;
9667 scale_info->inv_scale = (stbir_real) scale_info->scale_denominator / (stbir_real) scale_info->scale_numerator;
9668 }
9669
9670 // exact tap positions when the shift is an integer ( n0 x output_sub / ( n1 - n0 ), plus the clip ) and the fraction wasn't approximated.
9671 // the numerators are ( 2o + 1 + 2 shift ) D and ( 2i + 1 ) N: with sizes up to 2^30 (the resizer's limit) and the
9672 // fraction's terms below 2^30 they stay below 2^62
9673 scale_info->positions_exact = 0;
9674 scale_info->shift_pixels = 0;
9675 if ( ( scale_info->scale_is_rational ) && ( !scale_info->approximated_drift ) && ( s_d > 0 ) && ( ( ( (stbir_int64)s_n0 * unclipped_output_sub_range ) % ( s_n1 - s_n0 ) ) == 0 ) &&
9676 ( scale_info->scale_numerator < ( 1 << 30 ) ) && ( scale_info->scale_denominator < ( 1 << 30 ) ) )
9677 {
9678 stbir_int64 shift = ( (stbir_int64)s_n0 * unclipped_output_sub_range ) / ( s_n1 - s_n0 ) + ( *output_offset - unclipped_output_offset );
9679 if ( ( shift > -( 1 << 30 ) ) && ( shift < ( 1 << 30 ) ) ) // the shift is in output pixels too (an output tile far past the image's own size)
9680 {
9681 scale_info->shift_pixels = (int) shift;
9682 scale_info->tap_scale_in = (stbir_real) 1.0 / ( (stbir_real) 2.0 * (stbir_real) scale_info->scale_numerator );
9683 scale_info->tap_scale_out = (stbir_real) 1.0 / ( (stbir_real) 2.0 * (stbir_real) scale_info->scale_denominator );
9684 scale_info->pixel_shift = (stbir_real) shift;
9685 scale_info->positions_exact = 1;
9686 }
9687 }
9688
9689 scale_info->input_full_size = input_full_range;
9690 scale_info->output_sub_size = output_sub_range;
9691
9692 return 1;
9693}
9694
9695
9696static void stbir__init_and_set_layout( STBIR_RESIZE * resize, stbir_pixel_layout pixel_layout, stbir_datatype data_type )
9697{
9698 resize->input_cb = 0;
9699 resize->output_cb = 0;
9700 resize->user_data = resize;
9701 resize->samplers = 0;
9702 resize->called_alloc = 0;
9703 resize->horizontal_filter = STBIR_FILTER_DEFAULT;
9704 resize->horizontal_filter_kernel = 0; resize->horizontal_filter_support = 0; resize->horizontal_filter_kernel_new = 0; resize->horizontal_filter_support_new = 0; resize->horizontal_filter_radius = 0;
9705 resize->vertical_filter = STBIR_FILTER_DEFAULT;
9706 resize->vertical_filter_kernel = 0; resize->vertical_filter_support = 0; resize->vertical_filter_kernel_new = 0; resize->vertical_filter_support_new = 0; resize->vertical_filter_radius = 0;
9707 resize->horizontal_edge = STBIR_EDGE_CLAMP;
9708 resize->vertical_edge = STBIR_EDGE_CLAMP;
9709 resize->input_s0 = 0; resize->input_t0 = 0; resize->input_s1 = 1; resize->input_t1 = 1;
9710 resize->input_s_n0 = 0; resize->input_s_n1 = 1; resize->input_s_d = 1;
9711 resize->input_t_n0 = 0; resize->input_t_n1 = 1; resize->input_t_d = 1;
9712 resize->output_subx = 0; resize->output_suby = 0; resize->output_subw = resize->output_w; resize->output_subh = resize->output_h;
9713 resize->input_data_type = data_type;
9714 resize->output_data_type = data_type;
9715 resize->input_pixel_layout_public = pixel_layout;
9716 resize->output_pixel_layout_public = pixel_layout;
9717 resize->needs_rebuild = 1;
9718}
9719
9720STBIRDEF void stbir_resize_init( STBIR_RESIZE * resize,
9721 const void *input_pixels, int input_w, int input_h, stbir_int64 input_stride_in_bytes, // stride can be zero
9722 void *output_pixels, int output_w, int output_h, stbir_int64 output_stride_in_bytes, // stride can be zero
9723 stbir_pixel_layout pixel_layout, stbir_datatype data_type )
9724{
9725 resize->input_pixels = input_pixels;
9726 resize->input_w = input_w;
9727 resize->input_h = input_h;
9728 resize->input_stride_in_bytes = input_stride_in_bytes;
9729 resize->output_pixels = output_pixels;
9730 resize->output_w = output_w;
9731 resize->output_h = output_h;
9732 resize->output_stride_in_bytes = output_stride_in_bytes;
9733 resize->fast_alpha = 0;
9734
9735 stbir__init_and_set_layout( resize, pixel_layout, data_type );
9736}
9737
9738// You can update parameters any time after resize_init
9739STBIRDEF void stbir_set_datatypes( STBIR_RESIZE * resize, stbir_datatype input_type, stbir_datatype output_type ) // by default, datatype from resize_init
9740{
9741 resize->input_data_type = input_type;
9742 resize->output_data_type = output_type;
9743 if ( ( resize->samplers ) && ( !resize->needs_rebuild ) )
9744 stbir__update_info_from_resize( resize->samplers, resize );
9745}
9746
9747STBIRDEF void stbir_set_pixel_callbacks( STBIR_RESIZE * resize, stbir_input_callback * input_cb, stbir_output_callback * output_cb ) // no callbacks by default
9748{
9749 resize->input_cb = input_cb;
9750 resize->output_cb = output_cb;
9751
9752 if ( ( resize->samplers ) && ( !resize->needs_rebuild ) )
9753 {
9754 resize->samplers->in_pixels_cb = input_cb;
9755 resize->samplers->out_pixels_cb = output_cb;
9756 }
9757}
9758
9759STBIRDEF void stbir_set_user_data( STBIR_RESIZE * resize, void * user_data ) // pass back STBIR_RESIZE* by default
9760{
9761 resize->user_data = user_data;
9762 if ( ( resize->samplers ) && ( !resize->needs_rebuild ) )
9763 resize->samplers->user_data = user_data;
9764}
9765
9766STBIRDEF void stbir_set_buffer_ptrs( STBIR_RESIZE * resize, const void * input_pixels, stbir_int64 input_stride_in_bytes, void * output_pixels, stbir_int64 output_stride_in_bytes )
9767{
9768 resize->input_pixels = input_pixels;
9769 resize->input_stride_in_bytes = input_stride_in_bytes;
9770 resize->output_pixels = output_pixels;
9771 resize->output_stride_in_bytes = output_stride_in_bytes;
9772 if ( ( resize->samplers ) && ( !resize->needs_rebuild ) )
9773 stbir__update_info_from_resize( resize->samplers, resize );
9774}
9775
9776
9777STBIRDEF int stbir_set_edgemodes( STBIR_RESIZE * resize, stbir_edge horizontal_edge, stbir_edge vertical_edge ) // CLAMP by default
9778{
9779 resize->horizontal_edge = horizontal_edge;
9780 resize->vertical_edge = vertical_edge;
9781 resize->needs_rebuild = 1;
9782 return 1;
9783}
9784
9785STBIRDEF int stbir_set_filters( STBIR_RESIZE * resize, stbir_filter horizontal_filter, stbir_filter vertical_filter ) // STBIR_DEFAULT_FILTER_UPSAMPLE/DOWNSAMPLE by default
9786{
9787 resize->horizontal_filter = horizontal_filter;
9788 resize->vertical_filter = vertical_filter;
9789 resize->needs_rebuild = 1;
9790 return 1;
9791}
9792
9793STBIRDEF int stbir_set_filter_callbacks_new( STBIR_RESIZE * resize, stbir__kernel_callback_new * horizontal_filter, stbir__support_callback_new * horizontal_support, stbir__radius_callback_new * horizontal_radius, stbir__kernel_callback_new * vertical_filter, stbir__support_callback_new * vertical_support, stbir__radius_callback_new * vertical_radius )
9794{
9795 // a filter's extent is given one way: a support (filter pixels) or a radius (input pixels), not both
9796 if ( ( horizontal_support && horizontal_radius ) || ( vertical_support && vertical_radius ) )
9797 return 0;
9798 resize->horizontal_filter_kernel_new = horizontal_filter;
9799 resize->horizontal_filter_support_new = horizontal_support;
9800 resize->horizontal_filter_radius = horizontal_radius;
9801 resize->horizontal_filter_kernel = 0; // (the old form's, if it was set before)
9802 resize->horizontal_filter_support = 0;
9803 resize->vertical_filter_kernel_new = vertical_filter;
9804 resize->vertical_filter_support_new = vertical_support;
9805 resize->vertical_filter_radius = vertical_radius;
9806 resize->vertical_filter_kernel = 0;
9807 resize->vertical_filter_support = 0;
9808 resize->needs_rebuild = 1;
9809 return 1;
9810}
9811
9812// the old form: the float kernels and supports are kept apart and the sampler calls them through a conversion
9813STBIRDEF int stbir_set_filter_callbacks( STBIR_RESIZE * resize, stbir__kernel_callback * horizontal_filter, stbir__support_callback * horizontal_support, stbir__kernel_callback * vertical_filter, stbir__support_callback * vertical_support )
9814{
9815 resize->horizontal_filter_kernel = horizontal_filter;
9816 resize->horizontal_filter_support = horizontal_support;
9817 resize->horizontal_filter_kernel_new = 0;
9818 resize->horizontal_filter_support_new = 0;
9819 resize->horizontal_filter_radius = 0;
9820 resize->vertical_filter_kernel = vertical_filter;
9821 resize->vertical_filter_support = vertical_support;
9822 resize->vertical_filter_kernel_new = 0;
9823 resize->vertical_filter_support_new = 0;
9824 resize->vertical_filter_radius = 0;
9825 resize->needs_rebuild = 1;
9826 return 1;
9827}
9828
9829STBIRDEF int stbir_set_pixel_layouts( STBIR_RESIZE * resize, stbir_pixel_layout input_pixel_layout, stbir_pixel_layout output_pixel_layout ) // sets new pixel layouts
9830{
9831 resize->input_pixel_layout_public = input_pixel_layout;
9832 resize->output_pixel_layout_public = output_pixel_layout;
9833 resize->needs_rebuild = 1;
9834 return 1;
9835}
9836
9837
9838STBIRDEF int stbir_set_non_pm_alpha_speed_over_quality( STBIR_RESIZE * resize, int non_pma_alpha_speed_over_quality ) // sets alpha speed
9839{
9840 resize->fast_alpha = non_pma_alpha_speed_over_quality;
9841 resize->needs_rebuild = 1;
9842 return 1;
9843}
9844
9845STBIRDEF int stbir_set_input_subrect( STBIR_RESIZE * resize, stbir_real s0, stbir_real t0, stbir_real s1, stbir_real t1 ) // sets input region (full region by default)
9846{
9847 resize->input_s0 = s0;
9848 resize->input_t0 = t0;
9849 resize->input_s1 = s1;
9850 resize->input_t1 = t1;
9851 resize->input_s_d = 0; // not in pixels: the fractions are worked out from the reals at build time
9852 resize->input_t_d = 0;
9853 resize->needs_rebuild = 1;
9854
9855 // are we inbounds? (a zero or negative area, or a region wholly past the image, is empty)
9856 if ( ( s1 <= (stbir_real)0.0 ) || ( s1 <= s0 ) ||
9857 ( t1 <= (stbir_real)0.0 ) || ( t1 <= t0 ) ||
9858 ( s0 >= (stbir_real)1.0 ) ||
9859 ( t0 >= (stbir_real)1.0 ) )
9860 return 0;
9861
9862 return 1;
9863}
9864
9865STBIRDEF int stbir_set_output_pixel_subrect( STBIR_RESIZE * resize, int subx, int suby, int subw, int subh ) // sets the output sub rect, in output pixels (the whole output by default)
9866{
9867 resize->output_subx = subx;
9868 resize->output_suby = suby;
9869 resize->output_subw = subw;
9870 resize->output_subh = subh;
9871 resize->needs_rebuild = 1;
9872
9873 // is there anything in it? (a negative size is an empty rect - the resize then has nothing to do)
9874 if ( ( subx >= resize->output_w ) || ( ( subx + subw ) <= 0 ) || ( suby >= resize->output_h ) || ( ( suby + subh ) <= 0 ) || ( subw <= 0 ) || ( subh <= 0 ) )
9875 return 0;
9876
9877 return 1;
9878}
9879
9880STBIRDEF int stbir_set_pixel_subrect( STBIR_RESIZE * resize, int subx, int suby, int subw, int subh ) // sets both regions (full regions by default)
9881{
9882 stbir_real s0, t0, s1, t1;
9883
9884 s0 = ( (stbir_real)subx ) / ( (stbir_real)resize->output_w );
9885 t0 = ( (stbir_real)suby ) / ( (stbir_real)resize->output_h );
9886 s1 = ( (stbir_real)(subx+subw) ) / ( (stbir_real)resize->output_w );
9887 t1 = ( (stbir_real)(suby+subh) ) / ( (stbir_real)resize->output_h );
9888
9889 resize->input_s0 = s0;
9890 resize->input_t0 = t0;
9891 resize->input_s1 = s1;
9892 resize->input_t1 = t1;
9893 resize->input_s_n0 = subx;
9894 resize->input_s_n1 = subx + subw;
9895 resize->input_s_d = resize->output_w;
9896 resize->input_t_n0 = suby;
9897 resize->input_t_n1 = suby + subh;
9898 resize->input_t_d = resize->output_h;
9899 resize->output_subx = subx;
9900 resize->output_suby = suby;
9901 resize->output_subw = subw;
9902 resize->output_subh = subh;
9903 resize->needs_rebuild = 1;
9904
9905 // is there anything in it? (a negative size is an empty rect - the resize then has nothing to do)
9906 if ( ( subx >= resize->output_w ) || ( ( subx + subw ) <= 0 ) || ( suby >= resize->output_h ) || ( ( suby + subh ) <= 0 ) || ( subw <= 0 ) || ( subh <= 0 ) )
9907 return 0;
9908
9909 return 1;
9910}
9911
9912static int stbir__perform_build( STBIR_RESIZE * resize, int splits )
9913{
9914 stbir__contributors conservative = { 0, 0 };
9915 stbir__sampler horizontal, vertical;
9916 int new_output_subx, new_output_suby;
9917 stbir__info * out_info;
9918 #ifdef STBIR_PROFILE
9919 stbir__info profile_infod; // used to contain building profile info before everything is allocated
9920 stbir__info * profile_info = &profile_infod;
9921 #endif
9922
9923 // have we already built the samplers?
9924 if ( resize->samplers )
9925 return 0;
9926
9927 #define STBIR_RETURN_ERROR_AND_ASSERT( exp ) STBIR_ASSERT( !(exp) ); if (exp) return 0;
9928 STBIR_RETURN_ERROR_AND_ASSERT( (unsigned)resize->horizontal_filter >= STBIR_FILTER_OTHER)
9929 STBIR_RETURN_ERROR_AND_ASSERT( (unsigned)resize->vertical_filter >= STBIR_FILTER_OTHER)
9930 #undef STBIR_RETURN_ERROR_AND_ASSERT
9931
9932 if ( splits <= 0 )
9933 return 0;
9934
9935 STBIR_PROFILE_BUILD_FIRST_START( build );
9936
9937 new_output_subx = resize->output_subx;
9938 new_output_suby = resize->output_suby;
9939
9940 // do horizontal clip and scale calcs (an empty region - zero or negative sizes, nothing in bounds - fails the build)
9941 if ( !stbir__calculate_region_transform( &horizontal.scale_info, resize->output_w, &new_output_subx, resize->output_subw, resize->input_w, resize->input_s0, resize->input_s1, resize->input_s_n0, resize->input_s_n1, resize->input_s_d ) )
9942 return 0;
9943
9944 // do vertical clip and scale calcs
9945 if ( !stbir__calculate_region_transform( &vertical.scale_info, resize->output_h, &new_output_suby, resize->output_subh, resize->input_h, resize->input_t0, resize->input_t1, resize->input_t_n0, resize->input_t_n1, resize->input_t_d ) )
9946 return 0;
9947
9948 // if nothing to do, just return
9949 if ( ( horizontal.scale_info.output_sub_size == 0 ) || ( vertical.scale_info.output_sub_size == 0 ) )
9950 return 0;
9951
9952 stbir__set_sampler(&horizontal, resize->horizontal_filter, resize->horizontal_filter_kernel_new, resize->horizontal_filter_support_new, resize->horizontal_filter_radius, resize->horizontal_filter_kernel, resize->horizontal_filter_support, resize->horizontal_edge, &horizontal.scale_info, 1, resize->user_data );
9953 stbir__get_conservative_extents( &horizontal, &conservative, resize->user_data );
9954 stbir__set_sampler(&vertical, resize->vertical_filter, resize->vertical_filter_kernel_new, resize->vertical_filter_support_new, resize->vertical_filter_radius, resize->vertical_filter_kernel, resize->vertical_filter_support, resize->vertical_edge, &vertical.scale_info, 0, resize->user_data );
9955
9956 // the split count can come down in the build (stbir__choose_split_grid): the caller gets the count it can use
9957
9958 STBIR_PROFILE_BUILD_START( alloc );
9959 out_info = stbir__alloc_internal_mem_and_build_samplers( &horizontal, &vertical, &conservative, resize->input_pixel_layout_public, resize->output_pixel_layout_public, splits, new_output_subx, new_output_suby, resize->fast_alpha, stbir__pixel_channels[ stbir__pixel_layout_convert_public_to_internal[ resize->input_pixel_layout_public ] ] * stbir__type_size[ resize->input_data_type ], resize->user_data STBIR_ONLY_PROFILE_BUILD_SET_INFO );
9960 STBIR_PROFILE_BUILD_END( alloc );
9961 STBIR_PROFILE_BUILD_END( build );
9962
9963 if ( out_info )
9964 {
9965 splits = out_info->splits;
9966 resize->splits = splits;
9967 resize->samplers = out_info;
9968 resize->needs_rebuild = 0;
9969 #ifdef STBIR_PROFILE
9970 STBIR_MEMCPY( &out_info->profile, &profile_infod.profile, sizeof( out_info->profile ) );
9971 #endif
9972
9973 // update anything that can be changed without recalcing samplers
9974 stbir__update_info_from_resize( out_info, resize );
9975
9976 return splits;
9977 }
9978
9979 return 0;
9980}
9981
9982STBIRDEF void stbir_free_samplers( STBIR_RESIZE * resize )
9983{
9984 if ( resize->samplers )
9985 {
9986 stbir__free_internal_mem( resize->samplers );
9987 resize->samplers = 0;
9988 resize->called_alloc = 0;
9989 }
9990}
9991
9992STBIRDEF int stbir_build_samplers_with_splits( STBIR_RESIZE * resize, int splits )
9993{
9994 if ( ( resize->samplers == 0 ) || ( resize->needs_rebuild ) )
9995 {
9996 if ( resize->samplers )
9997 stbir_free_samplers( resize );
9998
9999 resize->called_alloc = 1;
10000 return stbir__perform_build( resize, splits );
10001 }
10002
10003 STBIR_PROFILE_BUILD_CLEAR( resize->samplers );
10004
10005 return resize->splits; // already built: the count that build settled on (a new request isn't applied without a rebuild)
10006}
10007
10008STBIRDEF int stbir_build_samplers( STBIR_RESIZE * resize )
10009{
10010 return stbir_build_samplers_with_splits( resize, 1 );
10011}
10012
10013STBIRDEF int stbir_resize_extended( STBIR_RESIZE * resize )
10014{
10015 int result;
10016
10017 if ( ( resize->samplers == 0 ) || ( resize->needs_rebuild ) )
10018 {
10019 int alloc_state = resize->called_alloc; // remember allocated state
10020
10021 if ( resize->samplers )
10022 {
10023 stbir__free_internal_mem( resize->samplers );
10024 resize->samplers = 0;
10025 }
10026
10027 if ( !stbir_build_samplers( resize ) )
10028 return 0; // (an empty resize fails the build too, so it comes out of here as a failure)
10029
10030 resize->called_alloc = alloc_state;
10031 }
10032 else
10033 {
10034 // didn't build anything - clear it
10035 STBIR_PROFILE_BUILD_CLEAR( resize->samplers );
10036 }
10037
10038 // do resize
10039 result = stbir__perform_resize( resize->samplers, 0, resize->splits );
10040
10041 // if we alloced, then free
10042 if ( !resize->called_alloc )
10043 {
10044 stbir_free_samplers( resize );
10045 resize->samplers = 0;
10046 }
10047
10048 return result;
10049}
10050
10051STBIRDEF int stbir_resize_extended_split( STBIR_RESIZE * resize, int split_start, int split_count )
10052{
10053 STBIR_ASSERT( resize->samplers );
10054
10055 // you **must** build samplers first when using split resize (and not change the resize after)
10056 if ( ( resize->samplers == 0 ) || ( resize->needs_rebuild ) )
10057 return 0;
10058
10059 // if we're just doing the whole thing, call full
10060 if ( ( split_start == -1 ) || ( ( split_start == 0 ) && ( split_count == resize->splits ) ) )
10061 return stbir_resize_extended( resize );
10062
10063 if ( ( split_start >= resize->splits ) || ( split_start < 0 ) || ( ( split_start + split_count ) > resize->splits ) || ( split_count <= 0 ) )
10064 return 0;
10065
10066 // do resize
10067 return stbir__perform_resize( resize->samplers, split_start, split_count );
10068}
10069
10070
10071static void * stbir_quick_resize_helper( const void *input_pixels , int input_w , int input_h, stbir_int64 input_stride_in_bytes,
10072 void *output_pixels, int output_w, int output_h, stbir_int64 output_stride_in_bytes,
10073 stbir_pixel_layout pixel_layout, stbir_datatype data_type, stbir_edge edge, stbir_filter filter )
10074{
10075 STBIR_RESIZE resize;
10076 stbir_int64 scanline_output_in_bytes;
10077 stbir_int64 positive_output_stride_in_bytes;
10078 void * start_ptr;
10079 void * free_ptr;
10080
10081 scanline_output_in_bytes = (stbir_int64)output_w * stbir__type_size[ data_type ] * stbir__pixel_channels[ stbir__pixel_layout_convert_public_to_internal[ pixel_layout ] ];
10082 if ( scanline_output_in_bytes <= 0 )
10083 return 0;
10084
10085 // if zero stride, use scanline output
10086 if ( output_stride_in_bytes == 0 )
10087 output_stride_in_bytes = scanline_output_in_bytes;
10088
10089 // abs value for inverted images (negative pitches)
10090 positive_output_stride_in_bytes = output_stride_in_bytes;
10091 if ( positive_output_stride_in_bytes < 0 )
10092 positive_output_stride_in_bytes = -positive_output_stride_in_bytes;
10093
10094 // is the requested stride smaller than the scanline output? if so, just fail
10095 if ( positive_output_stride_in_bytes < scanline_output_in_bytes )
10096 return 0;
10097
10098 start_ptr = output_pixels;
10099 free_ptr = 0; // no free pointer, since they passed buffer to use
10100
10101 // did they pass a zero for the dest? if so, allocate the buffer
10102 if ( output_pixels == 0 )
10103 {
10104 size_t size;
10105 char * ptr;
10106
10107 size = (size_t)positive_output_stride_in_bytes * (size_t)output_h;
10108 if ( size == 0 )
10109 return 0;
10110
10111 ptr = (char*) STBIR_MALLOC( size, 0 );
10112 if ( ptr == 0 )
10113 return 0;
10114
10115 free_ptr = ptr;
10116
10117 // point at the last scanline, if they requested a flipped image
10118 if ( output_stride_in_bytes < 0 )
10119 start_ptr = ptr + ( (size_t)positive_output_stride_in_bytes * (size_t)( output_h - 1 ) );
10120 else
10121 start_ptr = ptr;
10122 }
10123
10124 // ok, now do the resize
10125 stbir_resize_init( &resize,
10126 input_pixels, input_w, input_h, input_stride_in_bytes,
10127 start_ptr, output_w, output_h, output_stride_in_bytes,
10128 pixel_layout, data_type );
10129
10130 resize.horizontal_edge = edge;
10131 resize.vertical_edge = edge;
10132 resize.horizontal_filter = filter;
10133 resize.vertical_filter = filter;
10134
10135 if ( !stbir_resize_extended( &resize ) )
10136 {
10137 if ( free_ptr )
10138 STBIR_FREE( free_ptr, 0 );
10139 return 0;
10140 }
10141
10142 return (free_ptr) ? free_ptr : start_ptr;
10143}
10144
10145
10146STBIRDEF unsigned char * stbir_resize_uint8_linear( const unsigned char *input_pixels , int input_w , int input_h, stbir_int64 input_stride_in_bytes,
10147 unsigned char *output_pixels, int output_w, int output_h, stbir_int64 output_stride_in_bytes,
10148 stbir_pixel_layout pixel_layout )
10149{
10150 return (unsigned char *) stbir_quick_resize_helper( input_pixels , input_w , input_h, input_stride_in_bytes,
10151 output_pixels, output_w, output_h, output_stride_in_bytes,
10152 pixel_layout, STBIR_TYPE_UINT8, STBIR_EDGE_CLAMP, STBIR_FILTER_DEFAULT );
10153}
10154
10155STBIRDEF unsigned char * stbir_resize_uint8_srgb( const unsigned char *input_pixels , int input_w , int input_h, stbir_int64 input_stride_in_bytes,
10156 unsigned char *output_pixels, int output_w, int output_h, stbir_int64 output_stride_in_bytes,
10157 stbir_pixel_layout pixel_layout )
10158{
10159 return (unsigned char *) stbir_quick_resize_helper( input_pixels , input_w , input_h, input_stride_in_bytes,
10160 output_pixels, output_w, output_h, output_stride_in_bytes,
10161 pixel_layout, STBIR_TYPE_UINT8_SRGB, STBIR_EDGE_CLAMP, STBIR_FILTER_DEFAULT );
10162}
10163
10164
10165STBIRDEF float * stbir_resize_float_linear( const float *input_pixels , int input_w , int input_h, stbir_int64 input_stride_in_bytes,
10166 float *output_pixels, int output_w, int output_h, stbir_int64 output_stride_in_bytes,
10167 stbir_pixel_layout pixel_layout )
10168{
10169 return (float *) stbir_quick_resize_helper( input_pixels , input_w , input_h, input_stride_in_bytes,
10170 output_pixels, output_w, output_h, output_stride_in_bytes,
10171 pixel_layout, STBIR_TYPE_FLOAT, STBIR_EDGE_CLAMP, STBIR_FILTER_DEFAULT );
10172}
10173
10174
10175STBIRDEF void * stbir_resize( const void *input_pixels , int input_w , int input_h, stbir_int64 input_stride_in_bytes,
10176 void *output_pixels, int output_w, int output_h, stbir_int64 output_stride_in_bytes,
10177 stbir_pixel_layout pixel_layout, stbir_datatype data_type,
10178 stbir_edge edge, stbir_filter filter )
10179{
10180 return (void *) stbir_quick_resize_helper( input_pixels , input_w , input_h, input_stride_in_bytes,
10181 output_pixels, output_w, output_h, output_stride_in_bytes,
10182 pixel_layout, data_type, edge, filter );
10183}
10184
10185#ifdef STBIR_PROFILE
10186
10187STBIRDEF void stbir_resize_build_profile_info( STBIR_PROFILE_INFO * info, STBIR_RESIZE const * resize )
10188{
10189 static char const * bdescriptions[6] = { "Building", "Allocating", "Horizontal sampler", "Vertical sampler", "Coefficient cleanup", "Scatter coefficients" } ;
10190 stbir__info* samp = resize->samplers;
10191 int i;
10192
10193 for( i = 0 ; i < (int)STBIR__ARRAY_SIZE( bdescriptions ) ; i++)
10194 info->clocks[i] = samp->profile.array[i+1];
10195
10196 info->total_clocks = samp->profile.named.total;
10197 info->descriptions = bdescriptions;
10198 info->count = STBIR__ARRAY_SIZE( bdescriptions );
10199}
10200
10201STBIRDEF void stbir_resize_split_profile_info( STBIR_PROFILE_INFO * info, STBIR_RESIZE const * resize, int split_start, int split_count )
10202{
10203 static char const * descriptions[7] = { "Looping", "Vertical sampling", "Horizontal sampling", "Scanline input", "Scanline output", "Alpha weighting", "Alpha unweighting" };
10204 stbir__per_split_info * split_info;
10205 int s, i;
10206
10207 if ( split_start == -1 )
10208 {
10209 split_start = 0;
10210 split_count = resize->samplers->splits;
10211 }
10212
10213 if ( ( split_start >= resize->splits ) || ( split_start < 0 ) || ( ( split_start + split_count ) > resize->splits ) || ( split_count <= 0 ) )
10214 {
10215 info->total_clocks = 0;
10216 info->descriptions = 0;
10217 info->count = 0;
10218 return;
10219 }
10220
10221 split_info = resize->samplers->split_info + split_start;
10222
10223 // sum up the profile from all the splits
10224 for( i = 0 ; i < (int)STBIR__ARRAY_SIZE( descriptions ) ; i++ )
10225 {
10226 stbir_uint64 sum = 0;
10227 for( s = 0 ; s < split_count ; s++ )
10228 sum += split_info[s].profile.array[i+1];
10229 info->clocks[i] = sum;
10230 }
10231
10232 info->total_clocks = 0;
10233 for( s = 0 ; s < split_count ; s++ )
10234 info->total_clocks += split_info[s].profile.named.total;
10235 info->descriptions = descriptions;
10236 info->count = STBIR__ARRAY_SIZE( descriptions );
10237}
10238
10239STBIRDEF void stbir_resize_extended_profile_info( STBIR_PROFILE_INFO * info, STBIR_RESIZE const * resize )
10240{
10241 stbir_resize_split_profile_info( info, resize, -1, 0 );
10242}
10243
10244#endif // STBIR_PROFILE
10245
10246#undef STBIR_BGR
10247#undef STBIR_1CHANNEL
10248#undef STBIR_2CHANNEL
10249#undef STBIR_RGB
10250#undef STBIR_RGBA
10251#undef STBIR_4CHANNEL
10252#undef STBIR_BGRA
10253#undef STBIR_ARGB
10254#undef STBIR_ABGR
10255#undef STBIR_RA
10256#undef STBIR_AR
10257#undef STBIR_RGBA_PM
10258#undef STBIR_BGRA_PM
10259#undef STBIR_ARGB_PM
10260#undef STBIR_ABGR_PM
10261#undef STBIR_RA_PM
10262#undef STBIR_AR_PM
10263
10264#endif // STB_IMAGE_RESIZE_IMPLEMENTATION
10265
10266#else // STB_IMAGE_RESIZE_HORIZONTALS&STB_IMAGE_RESIZE_DO_VERTICALS
10267
10268// we reinclude the header file to define all the horizontal functions
10269// specializing each function for the number of coeffs is 20-40% faster *OVERALL*
10270
10271// by including the header file again this way, we can still debug the functions
10272
10273#define STBIR_strs_join2( start, mid, end ) start##mid##end
10274#define STBIR_strs_join1( start, mid, end ) STBIR_strs_join2( start, mid, end )
10275
10276#define STBIR_strs_join24( start, mid1, mid2, end ) start##mid1##mid2##end
10277#define STBIR_strs_join14( start, mid1, mid2, end ) STBIR_strs_join24( start, mid1, mid2, end )
10278
10279#ifdef STB_IMAGE_RESIZE_DO_CODERS
10280
10281#ifdef stbir__decode_suffix
10282#define STBIR__CODER_NAME( name ) STBIR_strs_join1( name, _, stbir__decode_suffix )
10283#else
10284#define STBIR__CODER_NAME( name ) name
10285#endif
10286
10287#ifdef stbir__decode_swizzle
10288#define stbir__decode_simdf8_flip(reg) STBIR_strs_join1( STBIR_strs_join1( STBIR_strs_join1( STBIR_strs_join1( stbir__simdf8_0123to,stbir__decode_order0,stbir__decode_order1),stbir__decode_order2,stbir__decode_order3),stbir__decode_order0,stbir__decode_order1),stbir__decode_order2,stbir__decode_order3)(reg, reg)
10289#define stbir__decode_simdf4_flip(reg) STBIR_strs_join1( STBIR_strs_join1( stbir__simdf_0123to,stbir__decode_order0,stbir__decode_order1),stbir__decode_order2,stbir__decode_order3)(reg, reg)
10290#define stbir__encode_simdf8_unflip(reg) STBIR_strs_join1( STBIR_strs_join1( STBIR_strs_join1( STBIR_strs_join1( stbir__simdf8_0123to,stbir__encode_order0,stbir__encode_order1),stbir__encode_order2,stbir__encode_order3),stbir__encode_order0,stbir__encode_order1),stbir__encode_order2,stbir__encode_order3)(reg, reg)
10291#define stbir__encode_simdf4_unflip(reg) STBIR_strs_join1( STBIR_strs_join1( stbir__simdf_0123to,stbir__encode_order0,stbir__encode_order1),stbir__encode_order2,stbir__encode_order3)(reg, reg)
10292#define stbir__encode_simdf16_unflip(reg) STBIR_strs_join1( STBIR_strs_join1( stbir__simdf16_0123to,stbir__encode_order0,stbir__encode_order1),stbir__encode_order2,stbir__encode_order3)(reg, reg) // (one lane's order, the shuffle repeats it)
10293// u8 and u16 inputs are swizzled as bytes before the expansion when there is a byte shuffle
10294#ifdef STBIR_SIMD_BYTE_SHUFFLE
10295#define STBIR__SWZ8( p ) ( ( (p)*4 + stbir__decode_order0 ) | ( ( (p)*4 + stbir__decode_order1 ) << 8 ) | ( ( (p)*4 + stbir__decode_order2 ) << 16 ) | ( ( (p)*4 + stbir__decode_order3 ) << 24 ) )
10296#define STBIR__SWZ16( p, a, b ) ( ( 2*((p)*4 + (a)) ) | ( ( 2*((p)*4 + (a)) + 1 ) << 8 ) | ( ( 2*((p)*4 + (b)) ) << 16 ) | ( ( 2*((p)*4 + (b)) + 1 ) << 24 ) )
10297static STBIR__SIMDI_CONST4( STBIR__CODER_NAME( stbir__decode_bytes_swizzle ), STBIR__SWZ8(0), STBIR__SWZ8(1), STBIR__SWZ8(2), STBIR__SWZ8(3) );
10298static STBIR__SIMDI_CONST4( STBIR__CODER_NAME( stbir__decode_words_swizzle ), STBIR__SWZ16(0,stbir__decode_order0,stbir__decode_order1), STBIR__SWZ16(0,stbir__decode_order2,stbir__decode_order3), STBIR__SWZ16(1,stbir__decode_order0,stbir__decode_order1), STBIR__SWZ16(1,stbir__decode_order2,stbir__decode_order3) );
10299#define stbir__decode_simdi_flip_u8( ireg ) stbir__simdi_shuffle_bytes( ireg, ireg, STBIR__CONSTI( STBIR__CODER_NAME( stbir__decode_bytes_swizzle ) ) )
10300#define stbir__decode_simdi_flip_u16( ireg ) stbir__simdi_shuffle_bytes( ireg, ireg, STBIR__CONSTI( STBIR__CODER_NAME( stbir__decode_words_swizzle ) ) )
10301#define stbir__decode_simdf8_flip_i( reg )
10302#define stbir__decode_simdf4_flip_i( reg )
10303#define stbir__decode_simdf16_flip_i( reg )
10304#ifdef STBIR_SIMD16
10305#define stbir__decode_simdi8_flip_u16( ireg ) stbir__simdi8_shuffle_bytes( ireg, ireg, STBIR__CONSTI( STBIR__CODER_NAME( stbir__decode_words_swizzle ) ) ) // (the 16-wide u16 decoder loads 32 bytes: the same table in both lanes)
10306#endif
10307#else
10308#define stbir__decode_simdi_flip_u8( ireg )
10309#define stbir__decode_simdi_flip_u16( ireg )
10310#define stbir__decode_simdi8_flip_u16( ireg )
10311#define stbir__decode_simdf8_flip_i( reg ) stbir__decode_simdf8_flip( reg )
10312#define stbir__decode_simdf4_flip_i( reg ) stbir__decode_simdf4_flip( reg )
10313#define stbir__decode_simdf16_flip_i( reg ) STBIR_strs_join1( STBIR_strs_join1( stbir__simdf16_0123to,stbir__decode_order0,stbir__decode_order1),stbir__decode_order2,stbir__decode_order3)(reg, reg)
10314#endif
10315#else
10316#define stbir__decode_order0 0
10317#define stbir__decode_order1 1
10318#define stbir__decode_order2 2
10319#define stbir__decode_order3 3
10320#define stbir__encode_order0 0
10321#define stbir__encode_order1 1
10322#define stbir__encode_order2 2
10323#define stbir__encode_order3 3
10324#define stbir__decode_simdf8_flip(reg)
10325#define stbir__decode_simdf4_flip(reg)
10326#define stbir__decode_simdi_flip_u8( ireg )
10327#define stbir__decode_simdi_flip_u16( ireg )
10328#define stbir__decode_simdi8_flip_u16( ireg )
10329#define stbir__decode_simdf8_flip_i( reg )
10330#define stbir__decode_simdf4_flip_i( reg )
10331#define stbir__decode_simdf16_flip_i( reg )
10332#define stbir__encode_simdf8_unflip(reg)
10333#define stbir__encode_simdf4_unflip(reg)
10334#define stbir__encode_simdf16_unflip(reg)
10335#endif
10336
10337#if defined(STBIR_SIMD16)
10338#define stbir__encode_simdfX_unflip stbir__encode_simdf16_unflip
10339#elif defined(STBIR_SIMD8)
10340#define stbir__encode_simdfX_unflip stbir__encode_simdf8_unflip
10341#else
10342#define stbir__encode_simdfX_unflip stbir__encode_simdf4_unflip
10343#endif
10344
10345static float * STBIR__CODER_NAME( stbir__decode_uint8_linear_scaled )( float * decodep, ptrdiff_t width_times_channels, void const * inputp )
10346{
10347 float STBIR_STREAMOUT_PTR( * ) decode = decodep;
10348 float * decode_end = (float*) decode + width_times_channels;
10349 unsigned char const * input = (unsigned char const*)inputp;
10350
10351 #ifdef STBIR_SIMD
10352 unsigned char const * end_input_m16 = input + width_times_channels - 16;
10353 if ( width_times_channels >= 16 )
10354 {
10355 decode_end -= 16;
10356 STBIR_NO_UNROLL_LOOP_START_INF_FOR
10357 for(;;)
10358 {
10359 #if defined(STBIR_SIMD16)
10360 stbir__simdi i; stbir__simdi16 o;
10361 stbir__simdf16 of;
10362 STBIR_NO_UNROLL(decode);
10363 stbir__simdi_load( i, input );
10364 stbir__decode_simdi_flip_u8( i );
10365 stbir__simdi16_expand_u8_to_u32( o, i );
10366 stbir__simdi16_convert_i32_to_float( of, o );
10367 stbir__simdf16_mult( of, of, STBIR_max_uint8_as_float_inverted16 );
10368 stbir__decode_simdf16_flip_i( of );
10369 stbir__simdf16_store( decode, of );
10370 #elif defined(STBIR_SIMD8)
10371 stbir__simdi i; stbir__simdi8 o0,o1;
10372 stbir__simdf8 of0, of1;
10373 STBIR_NO_UNROLL(decode);
10374 stbir__simdi_load( i, input );
10375 stbir__decode_simdi_flip_u8( i );
10376 stbir__simdi8_expand_u8_to_u32( o0, o1, i );
10377 stbir__simdi8_convert_i32_to_float( of0, o0 );
10378 stbir__simdi8_convert_i32_to_float( of1, o1 );
10379 stbir__simdf8_mult( of0, of0, STBIR_max_uint8_as_float_inverted8);
10380 stbir__simdf8_mult( of1, of1, STBIR_max_uint8_as_float_inverted8);
10381 stbir__decode_simdf8_flip_i( of0 );
10382 stbir__decode_simdf8_flip_i( of1 );
10383 stbir__simdf8_store( decode + 0, of0 );
10384 stbir__simdf8_store( decode + 8, of1 );
10385 #else
10386 stbir__simdi i, o0, o1, o2, o3;
10387 stbir__simdf of0, of1, of2, of3;
10388 STBIR_NO_UNROLL(decode);
10389 stbir__simdi_load( i, input );
10390 stbir__decode_simdi_flip_u8( i );
10391 stbir__simdi_expand_u8_to_u32( o0,o1,o2,o3,i);
10392 stbir__simdi_convert_i32_to_float( of0, o0 );
10393 stbir__simdi_convert_i32_to_float( of1, o1 );
10394 stbir__simdi_convert_i32_to_float( of2, o2 );
10395 stbir__simdi_convert_i32_to_float( of3, o3 );
10396 stbir__simdf_mult( of0, of0, STBIR__CONSTF(STBIR_max_uint8_as_float_inverted) );
10397 stbir__simdf_mult( of1, of1, STBIR__CONSTF(STBIR_max_uint8_as_float_inverted) );
10398 stbir__simdf_mult( of2, of2, STBIR__CONSTF(STBIR_max_uint8_as_float_inverted) );
10399 stbir__simdf_mult( of3, of3, STBIR__CONSTF(STBIR_max_uint8_as_float_inverted) );
10400 stbir__decode_simdf4_flip_i( of0 );
10401 stbir__decode_simdf4_flip_i( of1 );
10402 stbir__decode_simdf4_flip_i( of2 );
10403 stbir__decode_simdf4_flip_i( of3 );
10404 stbir__simdf_store( decode + 0, of0 );
10405 stbir__simdf_store( decode + 4, of1 );
10406 stbir__simdf_store( decode + 8, of2 );
10407 stbir__simdf_store( decode + 12, of3 );
10408 #endif
10409 decode += 16;
10410 input += 16;
10411 if ( decode <= decode_end )
10412 continue;
10413 if ( decode == ( decode_end + 16 ) )
10414 break;
10415 decode = decode_end; // backup and do last couple
10416 input = end_input_m16;
10417 }
10418 return decode_end + 16;
10419 }
10420 #endif
10421
10422 // try to do blocks of 4 when you can
10423 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
10424 decode += 4;
10425 STBIR_SIMD_NO_UNROLL_LOOP_START
10426 while( decode <= decode_end )
10427 {
10428 STBIR_SIMD_NO_UNROLL(decode);
10429 decode[0-4] = ((float)(input[stbir__decode_order0])) * stbir__max_uint8_as_float_inverted;
10430 decode[1-4] = ((float)(input[stbir__decode_order1])) * stbir__max_uint8_as_float_inverted;
10431 decode[2-4] = ((float)(input[stbir__decode_order2])) * stbir__max_uint8_as_float_inverted;
10432 decode[3-4] = ((float)(input[stbir__decode_order3])) * stbir__max_uint8_as_float_inverted;
10433 decode += 4;
10434 input += 4;
10435 }
10436 decode -= 4;
10437 #endif
10438
10439 // do the remnants
10440 #if stbir__coder_min_num < 4
10441 STBIR_NO_UNROLL_LOOP_START
10442 while( decode < decode_end )
10443 {
10444 STBIR_NO_UNROLL(decode);
10445 decode[0] = ((float)(input[stbir__decode_order0])) * stbir__max_uint8_as_float_inverted;
10446 #if stbir__coder_min_num >= 2
10447 decode[1] = ((float)(input[stbir__decode_order1])) * stbir__max_uint8_as_float_inverted;
10448 #endif
10449 #if stbir__coder_min_num >= 3
10450 decode[2] = ((float)(input[stbir__decode_order2])) * stbir__max_uint8_as_float_inverted;
10451 #endif
10452 decode += stbir__coder_min_num;
10453 input += stbir__coder_min_num;
10454 }
10455 #endif
10456
10457 return decode_end;
10458}
10459
10460static void STBIR__CODER_NAME( stbir__encode_uint8_linear_scaled )( void * outputp, ptrdiff_t width_times_channels, float const * encode )
10461{
10462 unsigned char STBIR_SIMD_STREAMOUT_PTR( * ) output = (unsigned char *) outputp;
10463 unsigned char * end_output = ( (unsigned char *) output ) + width_times_channels;
10464
10465 #ifdef STBIR_SIMD
10466 // double width blocks while they fit, then the four-at-a-time and scalar loops below finish the row. (the encode can
10467 // be in place - with an output callback it converts into the float buffer - so no backing up to redo the last block)
10468 STBIR_SIMD_NO_UNROLL_LOOP_START
10469 while ( ( end_output - output ) >= stbir__simdfX_float_count*2 )
10470 {
10471 stbir__simdfX e0, e1;
10472 STBIR_SIMD_NO_UNROLL(encode);
10473 stbir__simdfX_madd_mem( e0, STBIR_simd_point5X, STBIR_max_uint8_as_floatX, encode );
10474 stbir__simdfX_madd_mem( e1, STBIR_simd_point5X, STBIR_max_uint8_as_floatX, encode+stbir__simdfX_float_count );
10475 stbir__encode_simdfX_unflip( e0 );
10476 stbir__encode_simdfX_unflip( e1 );
10477 #if defined(STBIR_SIMD16)
10478 {
10479 stbir__simdi8 i;
10480 stbir__simdf16_pack_to_32bytes( i, e0, e1 );
10481 stbir__simdi8_store( output, i );
10482 }
10483 #elif defined(STBIR_SIMD8)
10484 {
10485 stbir__simdi i;
10486 stbir__simdf8_pack_to_16bytes( i, e0, e1 );
10487 stbir__simdi_store( output, i );
10488 }
10489 #else
10490 {
10491 stbir__simdi i;
10492 stbir__simdf_pack_to_8bytes( i, e0, e1 );
10493 stbir__simdi_store2( output, i );
10494 }
10495 #endif
10496 encode += stbir__simdfX_float_count*2;
10497 output += stbir__simdfX_float_count*2;
10498 }
10499
10500 // try to do blocks of 4 when you can
10501 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
10502 output += 4;
10503 STBIR_NO_UNROLL_LOOP_START
10504 while( output <= end_output )
10505 {
10506 stbir__simdf e0;
10507 stbir__simdi i0;
10508 STBIR_NO_UNROLL(encode);
10509 stbir__simdf_load( e0, encode );
10510 stbir__simdf_madd( e0, STBIR__CONSTF(STBIR_simd_point5), STBIR__CONSTF(STBIR_max_uint8_as_float), e0 );
10511 stbir__encode_simdf4_unflip( e0 );
10512 stbir__simdf_pack_to_8bytes( i0, e0, e0 ); // only use first 4
10513 *(int*)(output-4) = stbir__simdi_to_int( i0 );
10514 output += 4;
10515 encode += 4;
10516 }
10517 output -= 4;
10518 #endif
10519
10520 // do the remnants
10521 #if stbir__coder_min_num < 4
10522 STBIR_NO_UNROLL_LOOP_START
10523 while( output < end_output )
10524 {
10525 // read the whole pixel before writing any of it - the encode can be in place, and a swizzle reads a channel after
10526 // the one it lands on has been overwritten
10527 stbir__simdf e0;
10528 #if stbir__coder_min_num >= 2
10529 stbir__simdf e1;
10530 #endif
10531 #if stbir__coder_min_num >= 3
10532 stbir__simdf e2;
10533 #endif
10534 STBIR_NO_UNROLL(encode);
10535 stbir__simdf_madd1_mem( e0, STBIR__CONSTF(STBIR_simd_point5), STBIR__CONSTF(STBIR_max_uint8_as_float), encode+stbir__encode_order0 );
10536 #if stbir__coder_min_num >= 2
10537 stbir__simdf_madd1_mem( e1, STBIR__CONSTF(STBIR_simd_point5), STBIR__CONSTF(STBIR_max_uint8_as_float), encode+stbir__encode_order1 );
10538 #endif
10539 #if stbir__coder_min_num >= 3
10540 stbir__simdf_madd1_mem( e2, STBIR__CONSTF(STBIR_simd_point5), STBIR__CONSTF(STBIR_max_uint8_as_float), encode+stbir__encode_order2 );
10541 #endif
10542 output[0] = stbir__simdf_convert_float_to_uint8( e0 );
10543 #if stbir__coder_min_num >= 2
10544 output[1] = stbir__simdf_convert_float_to_uint8( e1 );
10545 #endif
10546 #if stbir__coder_min_num >= 3
10547 output[2] = stbir__simdf_convert_float_to_uint8( e2 );
10548 #endif
10549 output += stbir__coder_min_num;
10550 encode += stbir__coder_min_num;
10551 }
10552 #endif
10553
10554 #else
10555
10556 // try to do blocks of 4 when you can
10557 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
10558 output += 4;
10559 while( output <= end_output )
10560 {
10561 float f0 = encode[stbir__encode_order0] * stbir__max_uint8_as_float + 0.5f;
10562 float f1 = encode[stbir__encode_order1] * stbir__max_uint8_as_float + 0.5f;
10563 float f2 = encode[stbir__encode_order2] * stbir__max_uint8_as_float + 0.5f;
10564 float f3 = encode[stbir__encode_order3] * stbir__max_uint8_as_float + 0.5f;
10565 STBIR_CLAMP(f0, 0, 255);
10566 STBIR_CLAMP(f1, 0, 255);
10567 STBIR_CLAMP(f2, 0, 255);
10568 STBIR_CLAMP(f3, 0, 255);
10569 output[0-4] = (unsigned char)f0;
10570 output[1-4] = (unsigned char)f1;
10571 output[2-4] = (unsigned char)f2;
10572 output[3-4] = (unsigned char)f3;
10573 output += 4;
10574 encode += 4;
10575 }
10576 output -= 4;
10577 #endif
10578
10579 // do the remnants
10580 #if stbir__coder_min_num < 4
10581 STBIR_NO_UNROLL_LOOP_START
10582 while( output < end_output )
10583 {
10584 float f0 = encode[stbir__encode_order0] * stbir__max_uint8_as_float + 0.5f;
10585 #if stbir__coder_min_num >= 2
10586 float f1 = encode[stbir__encode_order1] * stbir__max_uint8_as_float + 0.5f;
10587 #endif
10588 #if stbir__coder_min_num >= 3
10589 float f2 = encode[stbir__encode_order2] * stbir__max_uint8_as_float + 0.5f;
10590 #endif
10591 STBIR_NO_UNROLL(encode);
10592 STBIR_CLAMP(f0, 0, 255);
10593 output[0] = (unsigned char)f0;
10594 #if stbir__coder_min_num >= 2
10595 STBIR_CLAMP(f1, 0, 255);
10596 output[1] = (unsigned char)f1;
10597 #endif
10598 #if stbir__coder_min_num >= 3
10599 STBIR_CLAMP(f2, 0, 255);
10600 output[2] = (unsigned char)f2;
10601 #endif
10602 output += stbir__coder_min_num;
10603 encode += stbir__coder_min_num;
10604 }
10605 #endif
10606 #endif
10607}
10608
10609static float * STBIR__CODER_NAME(stbir__decode_uint8_linear)( float * decodep, ptrdiff_t width_times_channels, void const * inputp )
10610{
10611 float STBIR_STREAMOUT_PTR( * ) decode = decodep;
10612 float * decode_end = (float*) decode + width_times_channels;
10613 unsigned char const * input = (unsigned char const*)inputp;
10614
10615 #ifdef STBIR_SIMD
10616 unsigned char const * end_input_m16 = input + width_times_channels - 16;
10617 if ( width_times_channels >= 16 )
10618 {
10619 decode_end -= 16;
10620 STBIR_NO_UNROLL_LOOP_START_INF_FOR
10621 for(;;)
10622 {
10623 #if defined(STBIR_SIMD16)
10624 stbir__simdi i; stbir__simdi16 o;
10625 stbir__simdf16 of;
10626 STBIR_NO_UNROLL(decode);
10627 stbir__simdi_load( i, input );
10628 stbir__decode_simdi_flip_u8( i );
10629 stbir__simdi16_expand_u8_to_u32( o, i );
10630 stbir__simdi16_convert_i32_to_float( of, o );
10631 stbir__decode_simdf16_flip_i( of );
10632 stbir__simdf16_store( decode, of );
10633 #elif defined(STBIR_SIMD8)
10634 stbir__simdi i; stbir__simdi8 o0,o1;
10635 stbir__simdf8 of0, of1;
10636 STBIR_NO_UNROLL(decode);
10637 stbir__simdi_load( i, input );
10638 stbir__decode_simdi_flip_u8( i );
10639 stbir__simdi8_expand_u8_to_u32( o0, o1, i );
10640 stbir__simdi8_convert_i32_to_float( of0, o0 );
10641 stbir__simdi8_convert_i32_to_float( of1, o1 );
10642 stbir__decode_simdf8_flip_i( of0 );
10643 stbir__decode_simdf8_flip_i( of1 );
10644 stbir__simdf8_store( decode + 0, of0 );
10645 stbir__simdf8_store( decode + 8, of1 );
10646 #else
10647 stbir__simdi i, o0, o1, o2, o3;
10648 stbir__simdf of0, of1, of2, of3;
10649 STBIR_NO_UNROLL(decode);
10650 stbir__simdi_load( i, input );
10651 stbir__decode_simdi_flip_u8( i );
10652 stbir__simdi_expand_u8_to_u32( o0,o1,o2,o3,i);
10653 stbir__simdi_convert_i32_to_float( of0, o0 );
10654 stbir__simdi_convert_i32_to_float( of1, o1 );
10655 stbir__simdi_convert_i32_to_float( of2, o2 );
10656 stbir__simdi_convert_i32_to_float( of3, o3 );
10657 stbir__decode_simdf4_flip_i( of0 );
10658 stbir__decode_simdf4_flip_i( of1 );
10659 stbir__decode_simdf4_flip_i( of2 );
10660 stbir__decode_simdf4_flip_i( of3 );
10661 stbir__simdf_store( decode + 0, of0 );
10662 stbir__simdf_store( decode + 4, of1 );
10663 stbir__simdf_store( decode + 8, of2 );
10664 stbir__simdf_store( decode + 12, of3 );
10665#endif
10666 decode += 16;
10667 input += 16;
10668 if ( decode <= decode_end )
10669 continue;
10670 if ( decode == ( decode_end + 16 ) )
10671 break;
10672 decode = decode_end; // backup and do last couple
10673 input = end_input_m16;
10674 }
10675 return decode_end + 16;
10676 }
10677 #endif
10678
10679 // try to do blocks of 4 when you can
10680 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
10681 decode += 4;
10682 STBIR_SIMD_NO_UNROLL_LOOP_START
10683 while( decode <= decode_end )
10684 {
10685 STBIR_SIMD_NO_UNROLL(decode);
10686 decode[0-4] = ((float)(input[stbir__decode_order0]));
10687 decode[1-4] = ((float)(input[stbir__decode_order1]));
10688 decode[2-4] = ((float)(input[stbir__decode_order2]));
10689 decode[3-4] = ((float)(input[stbir__decode_order3]));
10690 decode += 4;
10691 input += 4;
10692 }
10693 decode -= 4;
10694 #endif
10695
10696 // do the remnants
10697 #if stbir__coder_min_num < 4
10698 STBIR_NO_UNROLL_LOOP_START
10699 while( decode < decode_end )
10700 {
10701 STBIR_NO_UNROLL(decode);
10702 decode[0] = ((float)(input[stbir__decode_order0]));
10703 #if stbir__coder_min_num >= 2
10704 decode[1] = ((float)(input[stbir__decode_order1]));
10705 #endif
10706 #if stbir__coder_min_num >= 3
10707 decode[2] = ((float)(input[stbir__decode_order2]));
10708 #endif
10709 decode += stbir__coder_min_num;
10710 input += stbir__coder_min_num;
10711 }
10712 #endif
10713 return decode_end;
10714}
10715
10716static void STBIR__CODER_NAME( stbir__encode_uint8_linear )( void * outputp, ptrdiff_t width_times_channels, float const * encode )
10717{
10718 unsigned char STBIR_SIMD_STREAMOUT_PTR( * ) output = (unsigned char *) outputp;
10719 unsigned char * end_output = ( (unsigned char *) output ) + width_times_channels;
10720
10721 #ifdef STBIR_SIMD
10722 // double width blocks while they fit, then the four-at-a-time and scalar loops below finish the row. (the encode can
10723 // be in place - with an output callback it converts into the float buffer - so no backing up to redo the last block)
10724 STBIR_SIMD_NO_UNROLL_LOOP_START
10725 while ( ( end_output - output ) >= stbir__simdfX_float_count*2 )
10726 {
10727 stbir__simdfX e0, e1;
10728 STBIR_SIMD_NO_UNROLL(encode);
10729 stbir__simdfX_add_mem( e0, STBIR_simd_point5X, encode );
10730 stbir__simdfX_add_mem( e1, STBIR_simd_point5X, encode+stbir__simdfX_float_count );
10731 stbir__encode_simdfX_unflip( e0 );
10732 stbir__encode_simdfX_unflip( e1 );
10733 #if defined(STBIR_SIMD16)
10734 {
10735 stbir__simdi8 i;
10736 stbir__simdf16_pack_to_32bytes( i, e0, e1 );
10737 stbir__simdi8_store( output, i );
10738 }
10739 #elif defined(STBIR_SIMD8)
10740 {
10741 stbir__simdi i;
10742 stbir__simdf8_pack_to_16bytes( i, e0, e1 );
10743 stbir__simdi_store( output, i );
10744 }
10745 #else
10746 {
10747 stbir__simdi i;
10748 stbir__simdf_pack_to_8bytes( i, e0, e1 );
10749 stbir__simdi_store2( output, i );
10750 }
10751 #endif
10752 encode += stbir__simdfX_float_count*2;
10753 output += stbir__simdfX_float_count*2;
10754 }
10755
10756 // try to do blocks of 4 when you can
10757 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
10758 output += 4;
10759 STBIR_NO_UNROLL_LOOP_START
10760 while( output <= end_output )
10761 {
10762 stbir__simdf e0;
10763 stbir__simdi i0;
10764 STBIR_NO_UNROLL(encode);
10765 stbir__simdf_load( e0, encode );
10766 stbir__simdf_add( e0, STBIR__CONSTF(STBIR_simd_point5), e0 );
10767 stbir__encode_simdf4_unflip( e0 );
10768 stbir__simdf_pack_to_8bytes( i0, e0, e0 ); // only use first 4
10769 *(int*)(output-4) = stbir__simdi_to_int( i0 );
10770 output += 4;
10771 encode += 4;
10772 }
10773 output -= 4;
10774 #endif
10775
10776 #else
10777
10778 // try to do blocks of 4 when you can
10779 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
10780 output += 4;
10781 while( output <= end_output )
10782 {
10783 float f0 = encode[stbir__encode_order0] + 0.5f;
10784 float f1 = encode[stbir__encode_order1] + 0.5f;
10785 float f2 = encode[stbir__encode_order2] + 0.5f;
10786 float f3 = encode[stbir__encode_order3] + 0.5f;
10787 STBIR_CLAMP(f0, 0, 255);
10788 STBIR_CLAMP(f1, 0, 255);
10789 STBIR_CLAMP(f2, 0, 255);
10790 STBIR_CLAMP(f3, 0, 255);
10791 output[0-4] = (unsigned char)f0;
10792 output[1-4] = (unsigned char)f1;
10793 output[2-4] = (unsigned char)f2;
10794 output[3-4] = (unsigned char)f3;
10795 output += 4;
10796 encode += 4;
10797 }
10798 output -= 4;
10799 #endif
10800
10801 #endif
10802
10803 // do the remnants
10804 #if stbir__coder_min_num < 4
10805 STBIR_NO_UNROLL_LOOP_START
10806 while( output < end_output )
10807 {
10808 float f0 = encode[stbir__encode_order0] + 0.5f;
10809 #if stbir__coder_min_num >= 2
10810 float f1 = encode[stbir__encode_order1] + 0.5f;
10811 #endif
10812 #if stbir__coder_min_num >= 3
10813 float f2 = encode[stbir__encode_order2] + 0.5f;
10814 #endif
10815 STBIR_NO_UNROLL(encode);
10816 STBIR_CLAMP(f0, 0, 255);
10817 output[0] = (unsigned char)f0;
10818 #if stbir__coder_min_num >= 2
10819 STBIR_CLAMP(f1, 0, 255);
10820 output[1] = (unsigned char)f1;
10821 #endif
10822 #if stbir__coder_min_num >= 3
10823 STBIR_CLAMP(f2, 0, 255);
10824 output[2] = (unsigned char)f2;
10825 #endif
10826 output += stbir__coder_min_num;
10827 encode += stbir__coder_min_num;
10828 }
10829 #endif
10830}
10831
10832static float * STBIR__CODER_NAME(stbir__decode_uint8_srgb)( float * decodep, ptrdiff_t width_times_channels, void const * inputp )
10833{
10834 float STBIR_STREAMOUT_PTR( * ) decode = decodep;
10835 float * decode_end = (float*) decode + width_times_channels;
10836 unsigned char const * input = (unsigned char const *)inputp;
10837
10838 // try to do blocks of 4 when you can
10839 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
10840 decode += 4;
10841 while( decode <= decode_end )
10842 {
10843 decode[0-4] = stbir__srgb_uchar_to_linear_float[ input[ stbir__decode_order0 ] ];
10844 decode[1-4] = stbir__srgb_uchar_to_linear_float[ input[ stbir__decode_order1 ] ];
10845 decode[2-4] = stbir__srgb_uchar_to_linear_float[ input[ stbir__decode_order2 ] ];
10846 decode[3-4] = stbir__srgb_uchar_to_linear_float[ input[ stbir__decode_order3 ] ];
10847 decode += 4;
10848 input += 4;
10849 }
10850 decode -= 4;
10851 #endif
10852
10853 // do the remnants
10854 #if stbir__coder_min_num < 4
10855 STBIR_NO_UNROLL_LOOP_START
10856 while( decode < decode_end )
10857 {
10858 STBIR_NO_UNROLL(decode);
10859 decode[0] = stbir__srgb_uchar_to_linear_float[ input[ stbir__decode_order0 ] ];
10860 #if stbir__coder_min_num >= 2
10861 decode[1] = stbir__srgb_uchar_to_linear_float[ input[ stbir__decode_order1 ] ];
10862 #endif
10863 #if stbir__coder_min_num >= 3
10864 decode[2] = stbir__srgb_uchar_to_linear_float[ input[ stbir__decode_order2 ] ];
10865 #endif
10866 decode += stbir__coder_min_num;
10867 input += stbir__coder_min_num;
10868 }
10869 #endif
10870 return decode_end;
10871}
10872
10873#define stbir__min_max_shift20( i, f ) \
10874 do { \
10875 stbir__simdf_max( f, f, stbir_simdf_casti(STBIR__CONSTI( STBIR_almost_zero )) ); \
10876 stbir__simdf_min( f, f, stbir_simdf_casti(STBIR__CONSTI( STBIR_almost_one )) ); \
10877 stbir__simdi_32shr( i, stbir_simdi_castf( f ), 20 ); \
10878 } while(0)
10879
10880#define stbir__scale_and_convert( i, f ) \
10881 do { \
10882 stbir__simdf_madd( f, STBIR__CONSTF( STBIR_simd_point5 ), STBIR__CONSTF( STBIR_max_uint8_as_float ), f ); \
10883 stbir__simdf_max( f, f, stbir__simdf_zeroP() ); \
10884 stbir__simdf_min( f, f, STBIR__CONSTF( STBIR_max_uint8_as_float ) ); \
10885 stbir__simdf_convert_float_to_i32( i, f ); \
10886 } while(0)
10887
10888#define stbir__linear_to_srgb_finish( i, f ) \
10889{ \
10890 stbir__simdi temp; \
10891 stbir__simdi_32shr( temp, stbir_simdi_castf( f ), 12 ) ; \
10892 stbir__simdi_and( temp, temp, STBIR__CONSTI(STBIR_mantissa_mask) ); \
10893 stbir__simdi_or( temp, temp, STBIR__CONSTI(STBIR_topscale) ); \
10894 stbir__simdi_16madd( i, i, temp ); \
10895 stbir__simdi_32shr( i, i, 16 ); \
10896}
10897
10898// look up sRGB bytes for 4 float lanes per register (indexed by the float bits shifted right 20, see fp32_to_srgb8_tab4)
10899#define stbir__simdi_srgb_lookup_one( v, temp ) \
10900 do { \
10901 temp.m128i_i128 = v; \
10902 temp.m128i_u32[0] = STBIR__SRGB8_TAB4( temp.m128i_i32[0] ); \
10903 temp.m128i_u32[1] = STBIR__SRGB8_TAB4( temp.m128i_i32[1] ); \
10904 temp.m128i_u32[2] = STBIR__SRGB8_TAB4( temp.m128i_i32[2] ); \
10905 temp.m128i_u32[3] = STBIR__SRGB8_TAB4( temp.m128i_i32[3] ); \
10906 v = temp.m128i_i128; \
10907 } while(0)
10908
10909#define stbir__simdi_srgb_lookup2( v0,v1 ) \
10910 do { \
10911 stbir__simdi_u32 temp0,temp1; \
10912 stbir__simdi_srgb_lookup_one( v0, temp0 ); \
10913 stbir__simdi_srgb_lookup_one( v1, temp1 ); \
10914 } while(0)
10915
10916#define stbir__simdi_srgb_lookup3( v0,v1,v2 ) \
10917 do { \
10918 stbir__simdi_u32 temp0,temp1,temp2; \
10919 stbir__simdi_srgb_lookup_one( v0, temp0 ); \
10920 stbir__simdi_srgb_lookup_one( v1, temp1 ); \
10921 stbir__simdi_srgb_lookup_one( v2, temp2 ); \
10922 } while(0)
10923
10924#define stbir__simdi_srgb_lookup4( v0,v1,v2,v3 ) \
10925 do { \
10926 stbir__simdi_u32 temp0,temp1,temp2,temp3; \
10927 stbir__simdi_srgb_lookup_one( v0, temp0 ); \
10928 stbir__simdi_srgb_lookup_one( v1, temp1 ); \
10929 stbir__simdi_srgb_lookup_one( v2, temp2 ); \
10930 stbir__simdi_srgb_lookup_one( v3, temp3 ); \
10931 } while(0)
10932
10933static void STBIR__CODER_NAME( stbir__encode_uint8_srgb )( void * outputp, ptrdiff_t width_times_channels, float const * encode )
10934{
10935 unsigned char STBIR_SIMD_STREAMOUT_PTR( * ) output = (unsigned char*) outputp;
10936 unsigned char * end_output = ( (unsigned char*) output ) + width_times_channels;
10937
10938 #ifdef STBIR_SIMD
10939
10940 // double width blocks while they fit, then the four-at-a-time and scalar loops below finish the row. (the encode can
10941 // be in place - with an output callback it converts into the float buffer - so no backing up to redo the last block)
10942 STBIR_SIMD_NO_UNROLL_LOOP_START
10943 while ( ( end_output - output ) >= 16 )
10944 {
10945 stbir__simdf f0, f1, f2, f3;
10946 stbir__simdi i0, i1, i2, i3;
10947 STBIR_SIMD_NO_UNROLL(encode);
10948
10949 stbir__simdf_load4_transposed( f0, f1, f2, f3, encode );
10950
10951 stbir__min_max_shift20( i0, f0 );
10952 stbir__min_max_shift20( i1, f1 );
10953 stbir__min_max_shift20( i2, f2 );
10954 stbir__min_max_shift20( i3, f3 );
10955
10956 stbir__simdi_srgb_lookup4( i0, i1, i2, i3 );
10957
10958 stbir__linear_to_srgb_finish( i0, f0 );
10959 stbir__linear_to_srgb_finish( i1, f1 );
10960 stbir__linear_to_srgb_finish( i2, f2 );
10961 stbir__linear_to_srgb_finish( i3, f3 );
10962
10963 stbir__interleave_pack_and_store_16_u8( output, STBIR_strs_join1(i, ,stbir__encode_order0), STBIR_strs_join1(i, ,stbir__encode_order1), STBIR_strs_join1(i, ,stbir__encode_order2), STBIR_strs_join1(i, ,stbir__encode_order3) );
10964
10965 encode += 16;
10966 output += 16;
10967 }
10968 #endif
10969
10970 // try to do blocks of 4 when you can
10971 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
10972 output += 4;
10973 STBIR_SIMD_NO_UNROLL_LOOP_START
10974 while ( output <= end_output )
10975 {
10976 float e0, e1, e2, e3;
10977 STBIR_SIMD_NO_UNROLL(encode);
10978
10979 e0 = encode[stbir__encode_order0];
10980 e1 = encode[stbir__encode_order1];
10981 e2 = encode[stbir__encode_order2];
10982 e3 = encode[stbir__encode_order3];
10983 output[0-4] = stbir__linear_to_srgb_uchar( e0 );
10984 output[1-4] = stbir__linear_to_srgb_uchar( e1 );
10985 output[2-4] = stbir__linear_to_srgb_uchar( e2 );
10986 output[3-4] = stbir__linear_to_srgb_uchar( e3 );
10987
10988 output += 4;
10989 encode += 4;
10990 }
10991 output -= 4;
10992 #endif
10993
10994 // do the remnants
10995 #if stbir__coder_min_num < 4
10996 STBIR_NO_UNROLL_LOOP_START
10997 while( output < end_output )
10998 {
10999 float e0;
11000 #if stbir__coder_min_num >= 2
11001 float e1;
11002 #endif
11003 #if stbir__coder_min_num >= 3
11004 float e2;
11005 #endif
11006 STBIR_NO_UNROLL(encode);
11007 e0 = encode[stbir__encode_order0];
11008 #if stbir__coder_min_num >= 2
11009 e1 = encode[stbir__encode_order1];
11010 #endif
11011 #if stbir__coder_min_num >= 3
11012 e2 = encode[stbir__encode_order2];
11013 #endif
11014 output[0] = stbir__linear_to_srgb_uchar( e0 );
11015 #if stbir__coder_min_num >= 2
11016 output[1] = stbir__linear_to_srgb_uchar( e1 );
11017 #endif
11018 #if stbir__coder_min_num >= 3
11019 output[2] = stbir__linear_to_srgb_uchar( e2 );
11020 #endif
11021 output += stbir__coder_min_num;
11022 encode += stbir__coder_min_num;
11023 }
11024 #endif
11025}
11026
11027#if ( stbir__coder_min_num == 4 ) || ( ( stbir__coder_min_num == 1 ) && ( !defined(stbir__decode_swizzle) ) )
11028
11029static float * STBIR__CODER_NAME(stbir__decode_uint8_srgb4_linearalpha)( float * decodep, ptrdiff_t width_times_channels, void const * inputp )
11030{
11031 float STBIR_STREAMOUT_PTR( * ) decode = decodep;
11032 float * decode_end = (float*) decode + width_times_channels;
11033 unsigned char const * input = (unsigned char const *)inputp;
11034
11035 do {
11036 decode[0] = stbir__srgb_uchar_to_linear_float[ input[stbir__decode_order0] ];
11037 decode[1] = stbir__srgb_uchar_to_linear_float[ input[stbir__decode_order1] ];
11038 decode[2] = stbir__srgb_uchar_to_linear_float[ input[stbir__decode_order2] ];
11039 decode[3] = ( (float) input[stbir__decode_order3] ) * stbir__max_uint8_as_float_inverted;
11040 input += 4;
11041 decode += 4;
11042 } while( decode < decode_end );
11043 return decode_end;
11044}
11045
11046
11047static void STBIR__CODER_NAME( stbir__encode_uint8_srgb4_linearalpha )( void * outputp, ptrdiff_t width_times_channels, float const * encode )
11048{
11049 unsigned char STBIR_SIMD_STREAMOUT_PTR( * ) output = (unsigned char*) outputp;
11050 unsigned char * end_output = ( (unsigned char*) output ) + width_times_channels;
11051
11052 #ifdef STBIR_SIMD
11053
11054 // double width blocks while they fit, then the four-at-a-time and scalar loops below finish the row. (the encode can
11055 // be in place - with an output callback it converts into the float buffer - so no backing up to redo the last block)
11056 STBIR_SIMD_NO_UNROLL_LOOP_START
11057 while ( ( end_output - output ) >= 16 )
11058 {
11059 stbir__simdf f0, f1, f2, f3;
11060 stbir__simdi i0, i1, i2, i3;
11061
11062 STBIR_SIMD_NO_UNROLL(encode);
11063 stbir__simdf_load4_transposed( f0, f1, f2, f3, encode );
11064
11065 stbir__min_max_shift20( i0, f0 );
11066 stbir__min_max_shift20( i1, f1 );
11067 stbir__min_max_shift20( i2, f2 );
11068 stbir__scale_and_convert( i3, f3 );
11069
11070 stbir__simdi_srgb_lookup3( i0, i1, i2 );
11071
11072 stbir__linear_to_srgb_finish( i0, f0 );
11073 stbir__linear_to_srgb_finish( i1, f1 );
11074 stbir__linear_to_srgb_finish( i2, f2 );
11075
11076 stbir__interleave_pack_and_store_16_u8( output, STBIR_strs_join1(i, ,stbir__encode_order0), STBIR_strs_join1(i, ,stbir__encode_order1), STBIR_strs_join1(i, ,stbir__encode_order2), STBIR_strs_join1(i, ,stbir__encode_order3) );
11077
11078 output += 16;
11079 encode += 16;
11080
11081 }
11082 #endif
11083
11084 // the SIMD loop above can have finished the row, so this has to test first
11085 STBIR_SIMD_NO_UNROLL_LOOP_START
11086 while ( output < end_output )
11087 {
11088 float f;
11089 STBIR_SIMD_NO_UNROLL(encode);
11090
11091 output[stbir__decode_order0] = stbir__linear_to_srgb_uchar( encode[0] );
11092 output[stbir__decode_order1] = stbir__linear_to_srgb_uchar( encode[1] );
11093 output[stbir__decode_order2] = stbir__linear_to_srgb_uchar( encode[2] );
11094
11095 f = encode[3] * stbir__max_uint8_as_float + 0.5f;
11096 STBIR_CLAMP(f, 0, 255);
11097 output[stbir__decode_order3] = (unsigned char) f;
11098
11099 output += 4;
11100 encode += 4;
11101 }
11102}
11103
11104#endif
11105
11106#if ( stbir__coder_min_num == 2 ) || ( ( stbir__coder_min_num == 1 ) && ( !defined(stbir__decode_swizzle) ) )
11107
11108static float * STBIR__CODER_NAME(stbir__decode_uint8_srgb2_linearalpha)( float * decodep, ptrdiff_t width_times_channels, void const * inputp )
11109{
11110 float STBIR_STREAMOUT_PTR( * ) decode = decodep;
11111 float * decode_end = (float*) decode + width_times_channels;
11112 unsigned char const * input = (unsigned char const *)inputp;
11113
11114 decode += 4;
11115 while( decode <= decode_end )
11116 {
11117 decode[0-4] = stbir__srgb_uchar_to_linear_float[ input[stbir__decode_order0] ];
11118 decode[1-4] = ( (float) input[stbir__decode_order1] ) * stbir__max_uint8_as_float_inverted;
11119 decode[2-4] = stbir__srgb_uchar_to_linear_float[ input[stbir__decode_order0+2] ];
11120 decode[3-4] = ( (float) input[stbir__decode_order1+2] ) * stbir__max_uint8_as_float_inverted;
11121 input += 4;
11122 decode += 4;
11123 }
11124 decode -= 4;
11125 if( decode < decode_end )
11126 {
11127 decode[0] = stbir__srgb_uchar_to_linear_float[ input[stbir__decode_order0] ];
11128 decode[1] = ( (float) input[stbir__decode_order1] ) * stbir__max_uint8_as_float_inverted;
11129 }
11130 return decode_end;
11131}
11132
11133static void STBIR__CODER_NAME( stbir__encode_uint8_srgb2_linearalpha )( void * outputp, ptrdiff_t width_times_channels, float const * encode )
11134{
11135 unsigned char STBIR_SIMD_STREAMOUT_PTR( * ) output = (unsigned char*) outputp;
11136 unsigned char * end_output = ( (unsigned char*) output ) + width_times_channels;
11137
11138 #ifdef STBIR_SIMD
11139
11140 // double width blocks while they fit, then the four-at-a-time and scalar loops below finish the row. (the encode can
11141 // be in place - with an output callback it converts into the float buffer - so no backing up to redo the last block)
11142 STBIR_SIMD_NO_UNROLL_LOOP_START
11143 while ( ( end_output - output ) >= 16 )
11144 {
11145 stbir__simdf f0, f1, f2, f3;
11146 stbir__simdi i0, i1, i2, i3;
11147
11148 STBIR_SIMD_NO_UNROLL(encode);
11149 stbir__simdf_load4_transposed( f0, f1, f2, f3, encode );
11150
11151 stbir__min_max_shift20( i0, f0 );
11152 stbir__scale_and_convert( i1, f1 );
11153 stbir__min_max_shift20( i2, f2 );
11154 stbir__scale_and_convert( i3, f3 );
11155
11156 stbir__simdi_srgb_lookup2( i0, i2 );
11157
11158 stbir__linear_to_srgb_finish( i0, f0 );
11159 stbir__linear_to_srgb_finish( i2, f2 );
11160
11161 stbir__interleave_pack_and_store_16_u8( output, STBIR_strs_join1(i, ,stbir__encode_order0), STBIR_strs_join1(i, ,stbir__encode_order1), STBIR_strs_join1(i, ,stbir__encode_order2), STBIR_strs_join1(i, ,stbir__encode_order3) );
11162
11163 output += 16;
11164 encode += 16;
11165 }
11166 #endif
11167
11168 // the SIMD loop above can have finished the row, so this has to test first
11169 STBIR_SIMD_NO_UNROLL_LOOP_START
11170 while ( output < end_output )
11171 {
11172 float f;
11173 STBIR_SIMD_NO_UNROLL(encode);
11174
11175 output[stbir__decode_order0] = stbir__linear_to_srgb_uchar( encode[0] );
11176
11177 f = encode[1] * stbir__max_uint8_as_float + 0.5f;
11178 STBIR_CLAMP(f, 0, 255);
11179 output[stbir__decode_order1] = (unsigned char) f;
11180
11181 output += 2;
11182 encode += 2;
11183 }
11184}
11185
11186#endif
11187
11188static float * STBIR__CODER_NAME(stbir__decode_uint16_linear_scaled)( float * decodep, ptrdiff_t width_times_channels, void const * inputp )
11189{
11190 float STBIR_STREAMOUT_PTR( * ) decode = decodep;
11191 float * decode_end = (float*) decode + width_times_channels;
11192 unsigned short const * input = (unsigned short const *)inputp;
11193
11194 #ifdef STBIR_SIMD
11195 // a block is 16 words at 16 wide (one 32 byte load), 8 otherwise
11196 unsigned short const * end_input_mb = input + width_times_channels - STBIR__U16_DECODE_BLOCK;
11197 if ( width_times_channels >= STBIR__U16_DECODE_BLOCK )
11198 {
11199 decode_end -= STBIR__U16_DECODE_BLOCK;
11200 STBIR_NO_UNROLL_LOOP_START_INF_FOR
11201 for(;;)
11202 {
11203 #if defined(STBIR_SIMD16)
11204 stbir__simdi8 i; stbir__simdi16 o;
11205 stbir__simdf16 of;
11206 STBIR_NO_UNROLL(decode);
11207 stbir__simdi8_load( i, input );
11208 stbir__decode_simdi8_flip_u16( i );
11209 stbir__simdi16_expand_u16_to_u32( o, i );
11210 stbir__simdi16_convert_i32_to_float( of, o );
11211 stbir__simdf16_mult( of, of, STBIR_max_uint16_as_float_inverted16 );
11212 stbir__decode_simdf16_flip_i( of );
11213 stbir__simdf16_store( decode, of );
11214 #elif defined(STBIR_SIMD8)
11215 stbir__simdi i; stbir__simdi8 o;
11216 stbir__simdf8 of;
11217 STBIR_NO_UNROLL(decode);
11218 stbir__simdi_load( i, input );
11219 stbir__decode_simdi_flip_u16( i );
11220 stbir__simdi8_expand_u16_to_u32( o, i );
11221 stbir__simdi8_convert_i32_to_float( of, o );
11222 stbir__simdf8_mult( of, of, STBIR_max_uint16_as_float_inverted8);
11223 stbir__decode_simdf8_flip_i( of );
11224 stbir__simdf8_store( decode + 0, of );
11225 #else
11226 stbir__simdi i, o0, o1;
11227 stbir__simdf of0, of1;
11228 STBIR_NO_UNROLL(decode);
11229 stbir__simdi_load( i, input );
11230 stbir__decode_simdi_flip_u16( i );
11231 stbir__simdi_expand_u16_to_u32( o0,o1,i );
11232 stbir__simdi_convert_i32_to_float( of0, o0 );
11233 stbir__simdi_convert_i32_to_float( of1, o1 );
11234 stbir__simdf_mult( of0, of0, STBIR__CONSTF(STBIR_max_uint16_as_float_inverted) );
11235 stbir__simdf_mult( of1, of1, STBIR__CONSTF(STBIR_max_uint16_as_float_inverted));
11236 stbir__decode_simdf4_flip_i( of0 );
11237 stbir__decode_simdf4_flip_i( of1 );
11238 stbir__simdf_store( decode + 0, of0 );
11239 stbir__simdf_store( decode + 4, of1 );
11240 #endif
11241 decode += STBIR__U16_DECODE_BLOCK;
11242 input += STBIR__U16_DECODE_BLOCK;
11243 if ( decode <= decode_end )
11244 continue;
11245 if ( decode == ( decode_end + STBIR__U16_DECODE_BLOCK ) )
11246 break;
11247 decode = decode_end; // backup and do last couple
11248 input = end_input_mb;
11249 }
11250 return decode_end + STBIR__U16_DECODE_BLOCK;
11251 }
11252 #endif
11253
11254 // try to do blocks of 4 when you can
11255 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
11256 decode += 4;
11257 STBIR_SIMD_NO_UNROLL_LOOP_START
11258 while( decode <= decode_end )
11259 {
11260 STBIR_SIMD_NO_UNROLL(decode);
11261 decode[0-4] = ((float)(input[stbir__decode_order0])) * stbir__max_uint16_as_float_inverted;
11262 decode[1-4] = ((float)(input[stbir__decode_order1])) * stbir__max_uint16_as_float_inverted;
11263 decode[2-4] = ((float)(input[stbir__decode_order2])) * stbir__max_uint16_as_float_inverted;
11264 decode[3-4] = ((float)(input[stbir__decode_order3])) * stbir__max_uint16_as_float_inverted;
11265 decode += 4;
11266 input += 4;
11267 }
11268 decode -= 4;
11269 #endif
11270
11271 // do the remnants
11272 #if stbir__coder_min_num < 4
11273 STBIR_NO_UNROLL_LOOP_START
11274 while( decode < decode_end )
11275 {
11276 STBIR_NO_UNROLL(decode);
11277 decode[0] = ((float)(input[stbir__decode_order0])) * stbir__max_uint16_as_float_inverted;
11278 #if stbir__coder_min_num >= 2
11279 decode[1] = ((float)(input[stbir__decode_order1])) * stbir__max_uint16_as_float_inverted;
11280 #endif
11281 #if stbir__coder_min_num >= 3
11282 decode[2] = ((float)(input[stbir__decode_order2])) * stbir__max_uint16_as_float_inverted;
11283 #endif
11284 decode += stbir__coder_min_num;
11285 input += stbir__coder_min_num;
11286 }
11287 #endif
11288 return decode_end;
11289}
11290
11291
11292static void STBIR__CODER_NAME(stbir__encode_uint16_linear_scaled)( void * outputp, ptrdiff_t width_times_channels, float const * encode )
11293{
11294 unsigned short STBIR_SIMD_STREAMOUT_PTR( * ) output = (unsigned short*) outputp;
11295 unsigned short * end_output = ( (unsigned short*) output ) + width_times_channels;
11296
11297 #ifdef STBIR_SIMD
11298 {
11299 // double width blocks while they fit, then the four-at-a-time and scalar loops below finish the row. (the encode can
11300 // be in place - with an output callback it converts into the float buffer - so no backing up to redo the last block)
11301 STBIR_SIMD_NO_UNROLL_LOOP_START
11302 while ( ( end_output - output ) >= stbir__simdfX_float_count*2 )
11303 {
11304 stbir__simdfX e0, e1;
11305 stbir__simdiX i;
11306 STBIR_SIMD_NO_UNROLL(encode);
11307 stbir__simdfX_madd_mem( e0, STBIR_simd_point5X, STBIR_max_uint16_as_floatX, encode );
11308 stbir__simdfX_madd_mem( e1, STBIR_simd_point5X, STBIR_max_uint16_as_floatX, encode+stbir__simdfX_float_count );
11309 stbir__encode_simdfX_unflip( e0 );
11310 stbir__encode_simdfX_unflip( e1 );
11311 stbir__simdfX_pack_to_words( i, e0, e1 );
11312 stbir__simdiX_store( output, i );
11313 encode += stbir__simdfX_float_count*2;
11314 output += stbir__simdfX_float_count*2;
11315 }
11316 }
11317
11318 // try to do blocks of 4 when you can
11319 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
11320 output += 4;
11321 STBIR_NO_UNROLL_LOOP_START
11322 while( output <= end_output )
11323 {
11324 stbir__simdf e;
11325 stbir__simdi i;
11326 STBIR_NO_UNROLL(encode);
11327 stbir__simdf_load( e, encode );
11328 stbir__simdf_madd( e, STBIR__CONSTF(STBIR_simd_point5), STBIR__CONSTF(STBIR_max_uint16_as_float), e );
11329 stbir__encode_simdf4_unflip( e );
11330 stbir__simdf_pack_to_8words( i, e, e ); // only use first 4
11331 stbir__simdi_store2( output-4, i );
11332 output += 4;
11333 encode += 4;
11334 }
11335 output -= 4;
11336 #endif
11337
11338 // do the remnants
11339 #if stbir__coder_min_num < 4
11340 STBIR_NO_UNROLL_LOOP_START
11341 while( output < end_output )
11342 {
11343 // read the whole pixel before writing any of it - the encode can be in place, and a swizzle reads a channel after
11344 // the one it lands on has been overwritten
11345 stbir__simdf e0;
11346 #if stbir__coder_min_num >= 2
11347 stbir__simdf e1;
11348 #endif
11349 #if stbir__coder_min_num >= 3
11350 stbir__simdf e2;
11351 #endif
11352 STBIR_NO_UNROLL(encode);
11353 stbir__simdf_madd1_mem( e0, STBIR__CONSTF(STBIR_simd_point5), STBIR__CONSTF(STBIR_max_uint16_as_float), encode+stbir__encode_order0 );
11354 #if stbir__coder_min_num >= 2
11355 stbir__simdf_madd1_mem( e1, STBIR__CONSTF(STBIR_simd_point5), STBIR__CONSTF(STBIR_max_uint16_as_float), encode+stbir__encode_order1 );
11356 #endif
11357 #if stbir__coder_min_num >= 3
11358 stbir__simdf_madd1_mem( e2, STBIR__CONSTF(STBIR_simd_point5), STBIR__CONSTF(STBIR_max_uint16_as_float), encode+stbir__encode_order2 );
11359 #endif
11360 output[0] = stbir__simdf_convert_float_to_short( e0 );
11361 #if stbir__coder_min_num >= 2
11362 output[1] = stbir__simdf_convert_float_to_short( e1 );
11363 #endif
11364 #if stbir__coder_min_num >= 3
11365 output[2] = stbir__simdf_convert_float_to_short( e2 );
11366 #endif
11367 output += stbir__coder_min_num;
11368 encode += stbir__coder_min_num;
11369 }
11370 #endif
11371
11372 #else
11373
11374 // try to do blocks of 4 when you can
11375 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
11376 output += 4;
11377 STBIR_SIMD_NO_UNROLL_LOOP_START
11378 while( output <= end_output )
11379 {
11380 // read the whole pixel before writing any of it - the encode can be in place, and a swizzle reads a channel after
11381 // the one it lands on has been overwritten
11382 float f0 = encode[stbir__encode_order0] * stbir__max_uint16_as_float + 0.5f;
11383 float f1 = encode[stbir__encode_order1] * stbir__max_uint16_as_float + 0.5f;
11384 float f2 = encode[stbir__encode_order2] * stbir__max_uint16_as_float + 0.5f;
11385 float f3 = encode[stbir__encode_order3] * stbir__max_uint16_as_float + 0.5f;
11386 STBIR_SIMD_NO_UNROLL(encode);
11387 STBIR_CLAMP(f0, 0, 65535); output[0-4] = (unsigned short)f0;
11388 STBIR_CLAMP(f1, 0, 65535); output[1-4] = (unsigned short)f1;
11389 STBIR_CLAMP(f2, 0, 65535); output[2-4] = (unsigned short)f2;
11390 STBIR_CLAMP(f3, 0, 65535); output[3-4] = (unsigned short)f3;
11391 output += 4;
11392 encode += 4;
11393 }
11394 output -= 4;
11395 #endif
11396
11397 // do the remnants
11398 #if stbir__coder_min_num < 4
11399 STBIR_NO_UNROLL_LOOP_START
11400 while( output < end_output )
11401 {
11402 float f0 = encode[stbir__encode_order0] * stbir__max_uint16_as_float + 0.5f;
11403 #if stbir__coder_min_num >= 2
11404 float f1 = encode[stbir__encode_order1] * stbir__max_uint16_as_float + 0.5f;
11405 #endif
11406 #if stbir__coder_min_num >= 3
11407 float f2 = encode[stbir__encode_order2] * stbir__max_uint16_as_float + 0.5f;
11408 #endif
11409 STBIR_NO_UNROLL(encode);
11410 STBIR_CLAMP(f0, 0, 65535); output[0] = (unsigned short)f0;
11411 #if stbir__coder_min_num >= 2
11412 STBIR_CLAMP(f1, 0, 65535); output[1] = (unsigned short)f1;
11413 #endif
11414 #if stbir__coder_min_num >= 3
11415 STBIR_CLAMP(f2, 0, 65535); output[2] = (unsigned short)f2;
11416 #endif
11417 output += stbir__coder_min_num;
11418 encode += stbir__coder_min_num;
11419 }
11420 #endif
11421 #endif
11422}
11423
11424static float * STBIR__CODER_NAME(stbir__decode_uint16_linear)( float * decodep, ptrdiff_t width_times_channels, void const * inputp )
11425{
11426 float STBIR_STREAMOUT_PTR( * ) decode = decodep;
11427 float * decode_end = (float*) decode + width_times_channels;
11428 unsigned short const * input = (unsigned short const *)inputp;
11429
11430 #ifdef STBIR_SIMD
11431 unsigned short const * end_input_mb = input + width_times_channels - STBIR__U16_DECODE_BLOCK;
11432 if ( width_times_channels >= STBIR__U16_DECODE_BLOCK )
11433 {
11434 decode_end -= STBIR__U16_DECODE_BLOCK;
11435 STBIR_NO_UNROLL_LOOP_START_INF_FOR
11436 for(;;)
11437 {
11438 #if defined(STBIR_SIMD16)
11439 stbir__simdi8 i; stbir__simdi16 o;
11440 stbir__simdf16 of;
11441 STBIR_NO_UNROLL(decode);
11442 stbir__simdi8_load( i, input );
11443 stbir__decode_simdi8_flip_u16( i );
11444 stbir__simdi16_expand_u16_to_u32( o, i );
11445 stbir__simdi16_convert_i32_to_float( of, o );
11446 stbir__decode_simdf16_flip_i( of );
11447 stbir__simdf16_store( decode, of );
11448 #elif defined(STBIR_SIMD8)
11449 stbir__simdi i; stbir__simdi8 o;
11450 stbir__simdf8 of;
11451 STBIR_NO_UNROLL(decode);
11452 stbir__simdi_load( i, input );
11453 stbir__decode_simdi_flip_u16( i );
11454 stbir__simdi8_expand_u16_to_u32( o, i );
11455 stbir__simdi8_convert_i32_to_float( of, o );
11456 stbir__decode_simdf8_flip_i( of );
11457 stbir__simdf8_store( decode + 0, of );
11458 #else
11459 stbir__simdi i, o0, o1;
11460 stbir__simdf of0, of1;
11461 STBIR_NO_UNROLL(decode);
11462 stbir__simdi_load( i, input );
11463 stbir__decode_simdi_flip_u16( i );
11464 stbir__simdi_expand_u16_to_u32( o0, o1, i );
11465 stbir__simdi_convert_i32_to_float( of0, o0 );
11466 stbir__simdi_convert_i32_to_float( of1, o1 );
11467 stbir__decode_simdf4_flip_i( of0 );
11468 stbir__decode_simdf4_flip_i( of1 );
11469 stbir__simdf_store( decode + 0, of0 );
11470 stbir__simdf_store( decode + 4, of1 );
11471 #endif
11472 decode += STBIR__U16_DECODE_BLOCK;
11473 input += STBIR__U16_DECODE_BLOCK;
11474 if ( decode <= decode_end )
11475 continue;
11476 if ( decode == ( decode_end + STBIR__U16_DECODE_BLOCK ) )
11477 break;
11478 decode = decode_end; // backup and do last couple
11479 input = end_input_mb;
11480 }
11481 return decode_end + STBIR__U16_DECODE_BLOCK;
11482 }
11483 #endif
11484
11485 // try to do blocks of 4 when you can
11486 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
11487 decode += 4;
11488 STBIR_SIMD_NO_UNROLL_LOOP_START
11489 while( decode <= decode_end )
11490 {
11491 STBIR_SIMD_NO_UNROLL(decode);
11492 decode[0-4] = ((float)(input[stbir__decode_order0]));
11493 decode[1-4] = ((float)(input[stbir__decode_order1]));
11494 decode[2-4] = ((float)(input[stbir__decode_order2]));
11495 decode[3-4] = ((float)(input[stbir__decode_order3]));
11496 decode += 4;
11497 input += 4;
11498 }
11499 decode -= 4;
11500 #endif
11501
11502 // do the remnants
11503 #if stbir__coder_min_num < 4
11504 STBIR_NO_UNROLL_LOOP_START
11505 while( decode < decode_end )
11506 {
11507 STBIR_NO_UNROLL(decode);
11508 decode[0] = ((float)(input[stbir__decode_order0]));
11509 #if stbir__coder_min_num >= 2
11510 decode[1] = ((float)(input[stbir__decode_order1]));
11511 #endif
11512 #if stbir__coder_min_num >= 3
11513 decode[2] = ((float)(input[stbir__decode_order2]));
11514 #endif
11515 decode += stbir__coder_min_num;
11516 input += stbir__coder_min_num;
11517 }
11518 #endif
11519 return decode_end;
11520}
11521
11522static void STBIR__CODER_NAME(stbir__encode_uint16_linear)( void * outputp, ptrdiff_t width_times_channels, float const * encode )
11523{
11524 unsigned short STBIR_SIMD_STREAMOUT_PTR( * ) output = (unsigned short*) outputp;
11525 unsigned short * end_output = ( (unsigned short*) output ) + width_times_channels;
11526
11527 #ifdef STBIR_SIMD
11528 {
11529 // double width blocks while they fit, then the four-at-a-time and scalar loops below finish the row. (the encode can
11530 // be in place - with an output callback it converts into the float buffer - so no backing up to redo the last block)
11531 STBIR_SIMD_NO_UNROLL_LOOP_START
11532 while ( ( end_output - output ) >= stbir__simdfX_float_count*2 )
11533 {
11534 stbir__simdfX e0, e1;
11535 stbir__simdiX i;
11536 STBIR_SIMD_NO_UNROLL(encode);
11537 stbir__simdfX_add_mem( e0, STBIR_simd_point5X, encode );
11538 stbir__simdfX_add_mem( e1, STBIR_simd_point5X, encode+stbir__simdfX_float_count );
11539 stbir__encode_simdfX_unflip( e0 );
11540 stbir__encode_simdfX_unflip( e1 );
11541 stbir__simdfX_pack_to_words( i, e0, e1 );
11542 stbir__simdiX_store( output, i );
11543 encode += stbir__simdfX_float_count*2;
11544 output += stbir__simdfX_float_count*2;
11545 }
11546 }
11547
11548 // try to do blocks of 4 when you can
11549 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
11550 output += 4;
11551 STBIR_NO_UNROLL_LOOP_START
11552 while( output <= end_output )
11553 {
11554 stbir__simdf e;
11555 stbir__simdi i;
11556 STBIR_NO_UNROLL(encode);
11557 stbir__simdf_load( e, encode );
11558 stbir__simdf_add( e, STBIR__CONSTF(STBIR_simd_point5), e );
11559 stbir__encode_simdf4_unflip( e );
11560 stbir__simdf_pack_to_8words( i, e, e ); // only use first 4
11561 stbir__simdi_store2( output-4, i );
11562 output += 4;
11563 encode += 4;
11564 }
11565 output -= 4;
11566 #endif
11567
11568 #else
11569
11570 // try to do blocks of 4 when you can
11571 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
11572 output += 4;
11573 STBIR_SIMD_NO_UNROLL_LOOP_START
11574 while( output <= end_output )
11575 {
11576 // read the whole pixel before writing any of it - the encode can be in place, and a swizzle reads a channel after
11577 // the one it lands on has been overwritten
11578 float f0 = encode[stbir__encode_order0] + 0.5f;
11579 float f1 = encode[stbir__encode_order1] + 0.5f;
11580 float f2 = encode[stbir__encode_order2] + 0.5f;
11581 float f3 = encode[stbir__encode_order3] + 0.5f;
11582 STBIR_SIMD_NO_UNROLL(encode);
11583 STBIR_CLAMP(f0, 0, 65535); output[0-4] = (unsigned short)f0;
11584 STBIR_CLAMP(f1, 0, 65535); output[1-4] = (unsigned short)f1;
11585 STBIR_CLAMP(f2, 0, 65535); output[2-4] = (unsigned short)f2;
11586 STBIR_CLAMP(f3, 0, 65535); output[3-4] = (unsigned short)f3;
11587 output += 4;
11588 encode += 4;
11589 }
11590 output -= 4;
11591 #endif
11592
11593 #endif
11594
11595 // do the remnants
11596 #if stbir__coder_min_num < 4
11597 STBIR_NO_UNROLL_LOOP_START
11598 while( output < end_output )
11599 {
11600 float f0 = encode[stbir__encode_order0] + 0.5f;
11601 #if stbir__coder_min_num >= 2
11602 float f1 = encode[stbir__encode_order1] + 0.5f;
11603 #endif
11604 #if stbir__coder_min_num >= 3
11605 float f2 = encode[stbir__encode_order2] + 0.5f;
11606 #endif
11607 STBIR_NO_UNROLL(encode);
11608 STBIR_CLAMP(f0, 0, 65535); output[0] = (unsigned short)f0;
11609 #if stbir__coder_min_num >= 2
11610 STBIR_CLAMP(f1, 0, 65535); output[1] = (unsigned short)f1;
11611 #endif
11612 #if stbir__coder_min_num >= 3
11613 STBIR_CLAMP(f2, 0, 65535); output[2] = (unsigned short)f2;
11614 #endif
11615 output += stbir__coder_min_num;
11616 encode += stbir__coder_min_num;
11617 }
11618 #endif
11619}
11620
11621static float * STBIR__CODER_NAME(stbir__decode_half_float_linear)( float * decodep, ptrdiff_t width_times_channels, void const * inputp )
11622{
11623 float STBIR_STREAMOUT_PTR( * ) decode = decodep;
11624 float * decode_end = (float*) decode + width_times_channels;
11625 stbir__FP16 const * input = (stbir__FP16 const *)inputp;
11626
11627 #ifdef STBIR_SIMD
11628 if ( width_times_channels >= 8 )
11629 {
11630 stbir__FP16 const * end_input_m8 = input + width_times_channels - 8;
11631 decode_end -= 8;
11632 STBIR_NO_UNROLL_LOOP_START_INF_FOR
11633 for(;;)
11634 {
11635 STBIR_NO_UNROLL(decode);
11636
11637 stbir__half_to_float_SIMD( decode, input );
11638 #ifdef stbir__decode_swizzle
11639 #ifdef STBIR_SIMD8
11640 {
11641 stbir__simdf8 of;
11642 stbir__simdf8_load( of, decode );
11643 stbir__decode_simdf8_flip( of );
11644 stbir__simdf8_store( decode, of );
11645 }
11646 #else
11647 {
11648 stbir__simdf of0,of1;
11649 stbir__simdf_load( of0, decode );
11650 stbir__simdf_load( of1, decode+4 );
11651 stbir__decode_simdf4_flip( of0 );
11652 stbir__decode_simdf4_flip( of1 );
11653 stbir__simdf_store( decode, of0 );
11654 stbir__simdf_store( decode+4, of1 );
11655 }
11656 #endif
11657 #endif
11658 decode += 8;
11659 input += 8;
11660 if ( decode <= decode_end )
11661 continue;
11662 if ( decode == ( decode_end + 8 ) )
11663 break;
11664 decode = decode_end; // backup and do last couple
11665 input = end_input_m8;
11666 }
11667 return decode_end + 8;
11668 }
11669 #endif
11670
11671 // try to do blocks of 4 when you can
11672 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
11673 decode += 4;
11674 STBIR_SIMD_NO_UNROLL_LOOP_START
11675 while( decode <= decode_end )
11676 {
11677 STBIR_SIMD_NO_UNROLL(decode);
11678 decode[0-4] = stbir__half_to_float(input[stbir__decode_order0]);
11679 decode[1-4] = stbir__half_to_float(input[stbir__decode_order1]);
11680 decode[2-4] = stbir__half_to_float(input[stbir__decode_order2]);
11681 decode[3-4] = stbir__half_to_float(input[stbir__decode_order3]);
11682 decode += 4;
11683 input += 4;
11684 }
11685 decode -= 4;
11686 #endif
11687
11688 // do the remnants
11689 #if stbir__coder_min_num < 4
11690 STBIR_NO_UNROLL_LOOP_START
11691 while( decode < decode_end )
11692 {
11693 STBIR_NO_UNROLL(decode);
11694 decode[0] = stbir__half_to_float(input[stbir__decode_order0]);
11695 #if stbir__coder_min_num >= 2
11696 decode[1] = stbir__half_to_float(input[stbir__decode_order1]);
11697 #endif
11698 #if stbir__coder_min_num >= 3
11699 decode[2] = stbir__half_to_float(input[stbir__decode_order2]);
11700 #endif
11701 decode += stbir__coder_min_num;
11702 input += stbir__coder_min_num;
11703 }
11704 #endif
11705 return decode_end;
11706}
11707
11708static void STBIR__CODER_NAME( stbir__encode_half_float_linear )( void * outputp, ptrdiff_t width_times_channels, float const * encode )
11709{
11710 stbir__FP16 STBIR_SIMD_STREAMOUT_PTR( * ) output = (stbir__FP16*) outputp;
11711 stbir__FP16 * end_output = ( (stbir__FP16*) output ) + width_times_channels;
11712
11713 #ifdef STBIR_SIMD
11714 // double width blocks while they fit, then the four-at-a-time and scalar loops below finish the row. (the encode can
11715 // be in place - with an output callback it converts into the float buffer - so no backing up to redo the last block)
11716 STBIR_SIMD_NO_UNROLL_LOOP_START
11717 while ( ( end_output - output ) >= 8 )
11718 {
11719 STBIR_SIMD_NO_UNROLL(encode);
11720 #ifdef stbir__decode_swizzle
11721 #ifdef STBIR_SIMD8
11722 {
11723 stbir__simdf8 of;
11724 stbir__simdf8_load( of, encode );
11725 stbir__encode_simdf8_unflip( of );
11726 stbir__float_to_half_SIMD( output, (float*)&of );
11727 }
11728 #else
11729 {
11730 stbir__simdf of[2];
11731 stbir__simdf_load( of[0], encode );
11732 stbir__simdf_load( of[1], encode+4 );
11733 stbir__encode_simdf4_unflip( of[0] );
11734 stbir__encode_simdf4_unflip( of[1] );
11735 stbir__float_to_half_SIMD( output, (float*)of );
11736 }
11737 #endif
11738 #else
11739 stbir__float_to_half_SIMD( output, encode );
11740 #endif
11741 encode += 8;
11742 output += 8;
11743 }
11744 #endif
11745
11746 // try to do blocks of 4 when you can
11747 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
11748 output += 4;
11749 STBIR_SIMD_NO_UNROLL_LOOP_START
11750 while( output <= end_output )
11751 {
11752 float e0, e1, e2, e3;
11753 STBIR_SIMD_NO_UNROLL(output);
11754 e0 = encode[stbir__encode_order0];
11755 e1 = encode[stbir__encode_order1];
11756 e2 = encode[stbir__encode_order2];
11757 e3 = encode[stbir__encode_order3];
11758 output[0-4] = stbir__float_to_half(e0);
11759 output[1-4] = stbir__float_to_half(e1);
11760 output[2-4] = stbir__float_to_half(e2);
11761 output[3-4] = stbir__float_to_half(e3);
11762 output += 4;
11763 encode += 4;
11764 }
11765 output -= 4;
11766 #endif
11767
11768 // do the remnants
11769 #if stbir__coder_min_num < 4
11770 STBIR_NO_UNROLL_LOOP_START
11771 while( output < end_output )
11772 {
11773 float e0;
11774 #if stbir__coder_min_num >= 2
11775 float e1;
11776 #endif
11777 #if stbir__coder_min_num >= 3
11778 float e2;
11779 #endif
11780 STBIR_NO_UNROLL(output);
11781 e0 = encode[stbir__encode_order0];
11782 #if stbir__coder_min_num >= 2
11783 e1 = encode[stbir__encode_order1];
11784 #endif
11785 #if stbir__coder_min_num >= 3
11786 e2 = encode[stbir__encode_order2];
11787 #endif
11788 output[0] = stbir__float_to_half(e0);
11789 #if stbir__coder_min_num >= 2
11790 output[1] = stbir__float_to_half(e1);
11791 #endif
11792 #if stbir__coder_min_num >= 3
11793 output[2] = stbir__float_to_half(e2);
11794 #endif
11795 output += stbir__coder_min_num;
11796 encode += stbir__coder_min_num;
11797 }
11798 #endif
11799}
11800
11801static float * STBIR__CODER_NAME(stbir__decode_float_linear)( float * decodep, ptrdiff_t width_times_channels, void const * inputp )
11802{
11803 #ifdef stbir__decode_swizzle
11804 float STBIR_STREAMOUT_PTR( * ) decode = decodep;
11805 float * decode_end = (float*) decode + width_times_channels;
11806 float const * input = (float const *)inputp;
11807
11808 #ifdef STBIR_SIMD
11809 if ( width_times_channels >= 16 )
11810 {
11811 float const * end_input_m16 = input + width_times_channels - 16;
11812 decode_end -= 16;
11813 STBIR_NO_UNROLL_LOOP_START_INF_FOR
11814 for(;;)
11815 {
11816 STBIR_NO_UNROLL(decode);
11817 #ifdef STBIR_SIMD8
11818 {
11819 stbir__simdf8 of0,of1;
11820 stbir__simdf8_load( of0, input );
11821 stbir__simdf8_load( of1, input+8 );
11822 stbir__decode_simdf8_flip( of0 );
11823 stbir__decode_simdf8_flip( of1 );
11824 stbir__simdf8_store( decode, of0 );
11825 stbir__simdf8_store( decode+8, of1 );
11826 }
11827 #else
11828 {
11829 stbir__simdf of0,of1,of2,of3;
11830 stbir__simdf_load( of0, input );
11831 stbir__simdf_load( of1, input+4 );
11832 stbir__simdf_load( of2, input+8 );
11833 stbir__simdf_load( of3, input+12 );
11834 stbir__decode_simdf4_flip( of0 );
11835 stbir__decode_simdf4_flip( of1 );
11836 stbir__decode_simdf4_flip( of2 );
11837 stbir__decode_simdf4_flip( of3 );
11838 stbir__simdf_store( decode, of0 );
11839 stbir__simdf_store( decode+4, of1 );
11840 stbir__simdf_store( decode+8, of2 );
11841 stbir__simdf_store( decode+12, of3 );
11842 }
11843 #endif
11844 decode += 16;
11845 input += 16;
11846 if ( decode <= decode_end )
11847 continue;
11848 if ( decode == ( decode_end + 16 ) )
11849 break;
11850 decode = decode_end; // backup and do last couple
11851 input = end_input_m16;
11852 }
11853 return decode_end + 16;
11854 }
11855 #endif
11856
11857 // try to do blocks of 4 when you can
11858 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
11859 decode += 4;
11860 STBIR_SIMD_NO_UNROLL_LOOP_START
11861 while( decode <= decode_end )
11862 {
11863 STBIR_SIMD_NO_UNROLL(decode);
11864 decode[0-4] = input[stbir__decode_order0];
11865 decode[1-4] = input[stbir__decode_order1];
11866 decode[2-4] = input[stbir__decode_order2];
11867 decode[3-4] = input[stbir__decode_order3];
11868 decode += 4;
11869 input += 4;
11870 }
11871 decode -= 4;
11872 #endif
11873
11874 // do the remnants
11875 #if stbir__coder_min_num < 4
11876 STBIR_NO_UNROLL_LOOP_START
11877 while( decode < decode_end )
11878 {
11879 STBIR_NO_UNROLL(decode);
11880 decode[0] = input[stbir__decode_order0];
11881 #if stbir__coder_min_num >= 2
11882 decode[1] = input[stbir__decode_order1];
11883 #endif
11884 #if stbir__coder_min_num >= 3
11885 decode[2] = input[stbir__decode_order2];
11886 #endif
11887 decode += stbir__coder_min_num;
11888 input += stbir__coder_min_num;
11889 }
11890 #endif
11891 return decode_end;
11892
11893 #else
11894
11895 if ( (void*)decodep != inputp )
11896 STBIR_MEMCPY( decodep, inputp, width_times_channels * sizeof( float ) );
11897
11898 return decodep + width_times_channels;
11899
11900 #endif
11901}
11902
11903static void STBIR__CODER_NAME( stbir__encode_float_linear )( void * outputp, ptrdiff_t width_times_channels, float const * encode )
11904{
11905 #if !defined( STBIR_FLOAT_HIGH_CLAMP ) && !defined(STBIR_FLOAT_LOW_CLAMP) && !defined(stbir__decode_swizzle)
11906
11907 if ( (void*)outputp != (void*) encode )
11908 STBIR_MEMCPY( outputp, encode, width_times_channels * sizeof( float ) );
11909
11910 #else
11911
11912 float STBIR_SIMD_STREAMOUT_PTR( * ) output = (float*) outputp;
11913 float * end_output = ( (float*) output ) + width_times_channels;
11914
11915 #ifdef STBIR_FLOAT_HIGH_CLAMP
11916 #define stbir_scalar_hi_clamp( v ) if ( v > STBIR_FLOAT_HIGH_CLAMP ) v = STBIR_FLOAT_HIGH_CLAMP;
11917 #else
11918 #define stbir_scalar_hi_clamp( v )
11919 #endif
11920 #ifdef STBIR_FLOAT_LOW_CLAMP
11921 #define stbir_scalar_lo_clamp( v ) if ( v < STBIR_FLOAT_LOW_CLAMP ) v = STBIR_FLOAT_LOW_CLAMP;
11922 #else
11923 #define stbir_scalar_lo_clamp( v )
11924 #endif
11925
11926 #ifdef STBIR_SIMD
11927
11928 #ifdef STBIR_FLOAT_HIGH_CLAMP
11929 const stbir__simdfX high_clamp = stbir__simdf_frepX(STBIR_FLOAT_HIGH_CLAMP);
11930 #endif
11931 #ifdef STBIR_FLOAT_LOW_CLAMP
11932 const stbir__simdfX low_clamp = stbir__simdf_frepX(STBIR_FLOAT_LOW_CLAMP);
11933 #endif
11934
11935 // do as many double width blocks as fit and let the four-at-a-time and scalar loops below finish the row. (with an
11936 // output callback we encode in place, so we can't use the usual trick of backing up and redoing the last block)
11937 STBIR_SIMD_NO_UNROLL_LOOP_START
11938 while ( ( end_output - output ) >= ( stbir__simdfX_float_count * 2 ) )
11939 {
11940 stbir__simdfX e0, e1;
11941 STBIR_SIMD_NO_UNROLL(encode);
11942 stbir__simdfX_load( e0, encode );
11943 stbir__simdfX_load( e1, encode+stbir__simdfX_float_count );
11944#ifdef STBIR_FLOAT_HIGH_CLAMP
11945 stbir__simdfX_min( e0, e0, high_clamp );
11946 stbir__simdfX_min( e1, e1, high_clamp );
11947#endif
11948#ifdef STBIR_FLOAT_LOW_CLAMP
11949 stbir__simdfX_max( e0, e0, low_clamp );
11950 stbir__simdfX_max( e1, e1, low_clamp );
11951#endif
11952 stbir__encode_simdfX_unflip( e0 );
11953 stbir__encode_simdfX_unflip( e1 );
11954 stbir__simdfX_store( output, e0 );
11955 stbir__simdfX_store( output+stbir__simdfX_float_count, e1 );
11956 encode += stbir__simdfX_float_count * 2;
11957 output += stbir__simdfX_float_count * 2;
11958 }
11959
11960 // try to do blocks of 4 when you can
11961 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
11962 output += 4;
11963 STBIR_NO_UNROLL_LOOP_START
11964 while( output <= end_output )
11965 {
11966 stbir__simdf e0;
11967 STBIR_NO_UNROLL(encode);
11968 stbir__simdf_load( e0, encode );
11969#ifdef STBIR_FLOAT_HIGH_CLAMP
11970 stbir__simdf_min( e0, e0, stbir__simdfX_cast_to_simdf4( high_clamp ) );
11971#endif
11972#ifdef STBIR_FLOAT_LOW_CLAMP
11973 stbir__simdf_max( e0, e0, stbir__simdfX_cast_to_simdf4( low_clamp ) );
11974#endif
11975 stbir__encode_simdf4_unflip( e0 );
11976 stbir__simdf_store( output-4, e0 );
11977 output += 4;
11978 encode += 4;
11979 }
11980 output -= 4;
11981 #endif
11982
11983 #else
11984
11985 // try to do blocks of 4 when you can
11986 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
11987 output += 4;
11988 STBIR_SIMD_NO_UNROLL_LOOP_START
11989 while( output <= end_output )
11990 {
11991 // read everything before writing anything - with an output callback we encode in place, so this is an in-place unflip
11992 float e0 = encode[ stbir__encode_order0 ];
11993 float e1 = encode[ stbir__encode_order1 ];
11994 float e2 = encode[ stbir__encode_order2 ];
11995 float e3 = encode[ stbir__encode_order3 ];
11996 STBIR_SIMD_NO_UNROLL(encode);
11997 stbir_scalar_hi_clamp( e0 );
11998 stbir_scalar_lo_clamp( e0 );
11999 stbir_scalar_hi_clamp( e1 );
12000 stbir_scalar_lo_clamp( e1 );
12001 stbir_scalar_hi_clamp( e2 );
12002 stbir_scalar_lo_clamp( e2 );
12003 stbir_scalar_hi_clamp( e3 );
12004 stbir_scalar_lo_clamp( e3 );
12005 output[0-4] = e0;
12006 output[1-4] = e1;
12007 output[2-4] = e2;
12008 output[3-4] = e3;
12009 output += 4;
12010 encode += 4;
12011 }
12012 output -= 4;
12013
12014 #endif
12015
12016 #endif
12017
12018 // do the remnants
12019 #if stbir__coder_min_num < 4
12020 STBIR_NO_UNROLL_LOOP_START
12021 while( output < end_output )
12022 {
12023 float e0 = encode[ stbir__encode_order0 ];
12024 #if stbir__coder_min_num >= 2
12025 float e1 = encode[ stbir__encode_order1 ];
12026 #endif
12027 #if stbir__coder_min_num >= 3
12028 float e2 = encode[ stbir__encode_order2 ];
12029 #endif
12030 STBIR_NO_UNROLL(encode);
12031 stbir_scalar_hi_clamp( e0 ); stbir_scalar_lo_clamp( e0 ); output[0] = e0;
12032 #if stbir__coder_min_num >= 2
12033 stbir_scalar_hi_clamp( e1 ); stbir_scalar_lo_clamp( e1 ); output[1] = e1;
12034 #endif
12035 #if stbir__coder_min_num >= 3
12036 stbir_scalar_hi_clamp( e2 ); stbir_scalar_lo_clamp( e2 ); output[2] = e2;
12037 #endif
12038 output += stbir__coder_min_num;
12039 encode += stbir__coder_min_num;
12040 }
12041 #endif
12042
12043 #endif
12044}
12045
12046#undef stbir__decode_suffix
12047#undef stbir__decode_simdf8_flip
12048#undef stbir__decode_simdf8_flip_i
12049#undef stbir__decode_simdf4_flip_i
12050#undef stbir__decode_simdi_flip_u8
12051#undef stbir__decode_simdi_flip_u16
12052#ifdef STBIR__SWZ8
12053#undef STBIR__SWZ8
12054#undef STBIR__SWZ16
12055#endif
12056#undef stbir__decode_simdf4_flip
12057#undef stbir__decode_order0
12058#undef stbir__decode_order1
12059#undef stbir__decode_order2
12060#undef stbir__decode_order3
12061#undef stbir__encode_order0
12062#undef stbir__encode_order1
12063#undef stbir__encode_order2
12064#undef stbir__encode_order3
12065#undef stbir__encode_simdf8_unflip
12066#undef stbir__encode_simdf16_unflip
12067#undef stbir__decode_simdf16_flip_i
12068#undef stbir__decode_simdi8_flip_u16
12069#undef stbir__encode_simdf4_unflip
12070#undef stbir__encode_simdfX_unflip
12071#undef STBIR__CODER_NAME
12072#undef stbir__coder_min_num
12073#undef stbir__decode_swizzle
12074#undef stbir_scalar_hi_clamp
12075#undef stbir_scalar_lo_clamp
12076#undef STB_IMAGE_RESIZE_DO_CODERS
12077
12078#elif defined( STB_IMAGE_RESIZE_DO_VERTICALS)
12079
12080#ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
12081#define STBIR_chans( start, end ) STBIR_strs_join14(start,STBIR__vertical_channels,end,_cont)
12082#else
12083#define STBIR_chans( start, end ) STBIR_strs_join1(start,STBIR__vertical_channels,end)
12084#endif
12085
12086#if STBIR__vertical_channels >= 1
12087#define stbIF0( code ) code
12088#else
12089#define stbIF0( code )
12090#endif
12091#if STBIR__vertical_channels >= 2
12092#define stbIF1( code ) code
12093#else
12094#define stbIF1( code )
12095#endif
12096#if STBIR__vertical_channels >= 3
12097#define stbIF2( code ) code
12098#else
12099#define stbIF2( code )
12100#endif
12101#if STBIR__vertical_channels >= 4
12102#define stbIF3( code ) code
12103#else
12104#define stbIF3( code )
12105#endif
12106#if STBIR__vertical_channels >= 5
12107#define stbIF4( code ) code
12108#else
12109#define stbIF4( code )
12110#endif
12111#if STBIR__vertical_channels >= 6
12112#define stbIF5( code ) code
12113#else
12114#define stbIF5( code )
12115#endif
12116#if STBIR__vertical_channels >= 7
12117#define stbIF6( code ) code
12118#else
12119#define stbIF6( code )
12120#endif
12121#if STBIR__vertical_channels >= 8
12122#define stbIF7( code ) code
12123#else
12124#define stbIF7( code )
12125#endif
12126
12127static void STBIR_chans( stbir__vertical_scatter_with_,_coeffs)( float ** outputs, float const * vertical_coefficients, float const * input, float const * input_end )
12128{
12129 stbIF0( float STBIR_SIMD_STREAMOUT_PTR( * ) output0 = outputs[0]; float c0s = vertical_coefficients[0]; )
12130 stbIF1( float STBIR_SIMD_STREAMOUT_PTR( * ) output1 = outputs[1]; float c1s = vertical_coefficients[1]; )
12131 stbIF2( float STBIR_SIMD_STREAMOUT_PTR( * ) output2 = outputs[2]; float c2s = vertical_coefficients[2]; )
12132 stbIF3( float STBIR_SIMD_STREAMOUT_PTR( * ) output3 = outputs[3]; float c3s = vertical_coefficients[3]; )
12133 stbIF4( float STBIR_SIMD_STREAMOUT_PTR( * ) output4 = outputs[4]; float c4s = vertical_coefficients[4]; )
12134 stbIF5( float STBIR_SIMD_STREAMOUT_PTR( * ) output5 = outputs[5]; float c5s = vertical_coefficients[5]; )
12135 stbIF6( float STBIR_SIMD_STREAMOUT_PTR( * ) output6 = outputs[6]; float c6s = vertical_coefficients[6]; )
12136 stbIF7( float STBIR_SIMD_STREAMOUT_PTR( * ) output7 = outputs[7]; float c7s = vertical_coefficients[7]; )
12137
12138 #ifdef STBIR_SIMD
12139 {
12140 stbIF0(stbir__simdfX c0 = stbir__simdf_frepX( c0s ); )
12141 stbIF1(stbir__simdfX c1 = stbir__simdf_frepX( c1s ); )
12142 stbIF2(stbir__simdfX c2 = stbir__simdf_frepX( c2s ); )
12143 stbIF3(stbir__simdfX c3 = stbir__simdf_frepX( c3s ); )
12144 stbIF4(stbir__simdfX c4 = stbir__simdf_frepX( c4s ); )
12145 stbIF5(stbir__simdfX c5 = stbir__simdf_frepX( c5s ); )
12146 stbIF6(stbir__simdfX c6 = stbir__simdf_frepX( c6s ); )
12147 stbIF7(stbir__simdfX c7 = stbir__simdf_frepX( c7s ); )
12148 STBIR_SIMD_NO_UNROLL_LOOP_START
12149 while ( ( (char*)input_end - (char*) input ) >= (16*stbir__simdfX_float_count) )
12150 {
12151 stbir__simdfX o0, o1, o2, o3, r0, r1, r2, r3;
12152 STBIR_SIMD_NO_UNROLL(output0);
12153
12154 stbir__simdfX_load( r0, input ); stbir__simdfX_load( r1, input+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input+(3*stbir__simdfX_float_count) );
12155
12156 #ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
12157 stbIF0( stbir__simdfX_load( o0, output0 ); stbir__simdfX_load( o1, output0+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output0+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( o3, output0+(3*stbir__simdfX_float_count) );
12158 stbir__simdfX_madd( o0, o0, r0, c0 ); stbir__simdfX_madd( o1, o1, r1, c0 ); stbir__simdfX_madd( o2, o2, r2, c0 ); stbir__simdfX_madd( o3, o3, r3, c0 );
12159 stbir__simdfX_store( output0, o0 ); stbir__simdfX_store( output0+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output0+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output0+(3*stbir__simdfX_float_count), o3 ); )
12160 stbIF1( stbir__simdfX_load( o0, output1 ); stbir__simdfX_load( o1, output1+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output1+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( o3, output1+(3*stbir__simdfX_float_count) );
12161 stbir__simdfX_madd( o0, o0, r0, c1 ); stbir__simdfX_madd( o1, o1, r1, c1 ); stbir__simdfX_madd( o2, o2, r2, c1 ); stbir__simdfX_madd( o3, o3, r3, c1 );
12162 stbir__simdfX_store( output1, o0 ); stbir__simdfX_store( output1+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output1+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output1+(3*stbir__simdfX_float_count), o3 ); )
12163 stbIF2( stbir__simdfX_load( o0, output2 ); stbir__simdfX_load( o1, output2+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output2+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( o3, output2+(3*stbir__simdfX_float_count) );
12164 stbir__simdfX_madd( o0, o0, r0, c2 ); stbir__simdfX_madd( o1, o1, r1, c2 ); stbir__simdfX_madd( o2, o2, r2, c2 ); stbir__simdfX_madd( o3, o3, r3, c2 );
12165 stbir__simdfX_store( output2, o0 ); stbir__simdfX_store( output2+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output2+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output2+(3*stbir__simdfX_float_count), o3 ); )
12166 stbIF3( stbir__simdfX_load( o0, output3 ); stbir__simdfX_load( o1, output3+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output3+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( o3, output3+(3*stbir__simdfX_float_count) );
12167 stbir__simdfX_madd( o0, o0, r0, c3 ); stbir__simdfX_madd( o1, o1, r1, c3 ); stbir__simdfX_madd( o2, o2, r2, c3 ); stbir__simdfX_madd( o3, o3, r3, c3 );
12168 stbir__simdfX_store( output3, o0 ); stbir__simdfX_store( output3+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output3+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output3+(3*stbir__simdfX_float_count), o3 ); )
12169 stbIF4( stbir__simdfX_load( o0, output4 ); stbir__simdfX_load( o1, output4+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output4+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( o3, output4+(3*stbir__simdfX_float_count) );
12170 stbir__simdfX_madd( o0, o0, r0, c4 ); stbir__simdfX_madd( o1, o1, r1, c4 ); stbir__simdfX_madd( o2, o2, r2, c4 ); stbir__simdfX_madd( o3, o3, r3, c4 );
12171 stbir__simdfX_store( output4, o0 ); stbir__simdfX_store( output4+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output4+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output4+(3*stbir__simdfX_float_count), o3 ); )
12172 stbIF5( stbir__simdfX_load( o0, output5 ); stbir__simdfX_load( o1, output5+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output5+(2*stbir__simdfX_float_count)); stbir__simdfX_load( o3, output5+(3*stbir__simdfX_float_count) );
12173 stbir__simdfX_madd( o0, o0, r0, c5 ); stbir__simdfX_madd( o1, o1, r1, c5 ); stbir__simdfX_madd( o2, o2, r2, c5 ); stbir__simdfX_madd( o3, o3, r3, c5 );
12174 stbir__simdfX_store( output5, o0 ); stbir__simdfX_store( output5+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output5+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output5+(3*stbir__simdfX_float_count), o3 ); )
12175 stbIF6( stbir__simdfX_load( o0, output6 ); stbir__simdfX_load( o1, output6+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output6+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( o3, output6+(3*stbir__simdfX_float_count) );
12176 stbir__simdfX_madd( o0, o0, r0, c6 ); stbir__simdfX_madd( o1, o1, r1, c6 ); stbir__simdfX_madd( o2, o2, r2, c6 ); stbir__simdfX_madd( o3, o3, r3, c6 );
12177 stbir__simdfX_store( output6, o0 ); stbir__simdfX_store( output6+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output6+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output6+(3*stbir__simdfX_float_count), o3 ); )
12178 stbIF7( stbir__simdfX_load( o0, output7 ); stbir__simdfX_load( o1, output7+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output7+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( o3, output7+(3*stbir__simdfX_float_count) );
12179 stbir__simdfX_madd( o0, o0, r0, c7 ); stbir__simdfX_madd( o1, o1, r1, c7 ); stbir__simdfX_madd( o2, o2, r2, c7 ); stbir__simdfX_madd( o3, o3, r3, c7 );
12180 stbir__simdfX_store( output7, o0 ); stbir__simdfX_store( output7+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output7+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output7+(3*stbir__simdfX_float_count), o3 ); )
12181 #else
12182 stbIF0( stbir__simdfX_mult( o0, r0, c0 ); stbir__simdfX_mult( o1, r1, c0 ); stbir__simdfX_mult( o2, r2, c0 ); stbir__simdfX_mult( o3, r3, c0 );
12183 stbir__simdfX_store( output0, o0 ); stbir__simdfX_store( output0+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output0+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output0+(3*stbir__simdfX_float_count), o3 ); )
12184 stbIF1( stbir__simdfX_mult( o0, r0, c1 ); stbir__simdfX_mult( o1, r1, c1 ); stbir__simdfX_mult( o2, r2, c1 ); stbir__simdfX_mult( o3, r3, c1 );
12185 stbir__simdfX_store( output1, o0 ); stbir__simdfX_store( output1+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output1+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output1+(3*stbir__simdfX_float_count), o3 ); )
12186 stbIF2( stbir__simdfX_mult( o0, r0, c2 ); stbir__simdfX_mult( o1, r1, c2 ); stbir__simdfX_mult( o2, r2, c2 ); stbir__simdfX_mult( o3, r3, c2 );
12187 stbir__simdfX_store( output2, o0 ); stbir__simdfX_store( output2+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output2+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output2+(3*stbir__simdfX_float_count), o3 ); )
12188 stbIF3( stbir__simdfX_mult( o0, r0, c3 ); stbir__simdfX_mult( o1, r1, c3 ); stbir__simdfX_mult( o2, r2, c3 ); stbir__simdfX_mult( o3, r3, c3 );
12189 stbir__simdfX_store( output3, o0 ); stbir__simdfX_store( output3+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output3+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output3+(3*stbir__simdfX_float_count), o3 ); )
12190 stbIF4( stbir__simdfX_mult( o0, r0, c4 ); stbir__simdfX_mult( o1, r1, c4 ); stbir__simdfX_mult( o2, r2, c4 ); stbir__simdfX_mult( o3, r3, c4 );
12191 stbir__simdfX_store( output4, o0 ); stbir__simdfX_store( output4+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output4+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output4+(3*stbir__simdfX_float_count), o3 ); )
12192 stbIF5( stbir__simdfX_mult( o0, r0, c5 ); stbir__simdfX_mult( o1, r1, c5 ); stbir__simdfX_mult( o2, r2, c5 ); stbir__simdfX_mult( o3, r3, c5 );
12193 stbir__simdfX_store( output5, o0 ); stbir__simdfX_store( output5+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output5+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output5+(3*stbir__simdfX_float_count), o3 ); )
12194 stbIF6( stbir__simdfX_mult( o0, r0, c6 ); stbir__simdfX_mult( o1, r1, c6 ); stbir__simdfX_mult( o2, r2, c6 ); stbir__simdfX_mult( o3, r3, c6 );
12195 stbir__simdfX_store( output6, o0 ); stbir__simdfX_store( output6+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output6+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output6+(3*stbir__simdfX_float_count), o3 ); )
12196 stbIF7( stbir__simdfX_mult( o0, r0, c7 ); stbir__simdfX_mult( o1, r1, c7 ); stbir__simdfX_mult( o2, r2, c7 ); stbir__simdfX_mult( o3, r3, c7 );
12197 stbir__simdfX_store( output7, o0 ); stbir__simdfX_store( output7+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output7+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output7+(3*stbir__simdfX_float_count), o3 ); )
12198 #endif
12199
12200 input += (4*stbir__simdfX_float_count);
12201 stbIF0( output0 += (4*stbir__simdfX_float_count); )
12202 stbIF1( output1 += (4*stbir__simdfX_float_count); )
12203 stbIF2( output2 += (4*stbir__simdfX_float_count); )
12204 stbIF3( output3 += (4*stbir__simdfX_float_count); )
12205 stbIF4( output4 += (4*stbir__simdfX_float_count); )
12206 stbIF5( output5 += (4*stbir__simdfX_float_count); )
12207 stbIF6( output6 += (4*stbir__simdfX_float_count); )
12208 stbIF7( output7 += (4*stbir__simdfX_float_count); )
12209 }
12210 STBIR_SIMD_NO_UNROLL_LOOP_START
12211 while ( ( (char*)input_end - (char*) input ) >= 16 )
12212 {
12213 stbir__simdf o0, r0;
12214 STBIR_SIMD_NO_UNROLL(output0);
12215
12216 stbir__simdf_load( r0, input );
12217
12218 #ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
12219 stbIF0( stbir__simdf_load( o0, output0 ); stbir__simdf_madd( o0, o0, r0, stbir__simdfX_cast_to_simdf4( c0 ) ); stbir__simdf_store( output0, o0 ); )
12220 stbIF1( stbir__simdf_load( o0, output1 ); stbir__simdf_madd( o0, o0, r0, stbir__simdfX_cast_to_simdf4( c1 ) ); stbir__simdf_store( output1, o0 ); )
12221 stbIF2( stbir__simdf_load( o0, output2 ); stbir__simdf_madd( o0, o0, r0, stbir__simdfX_cast_to_simdf4( c2 ) ); stbir__simdf_store( output2, o0 ); )
12222 stbIF3( stbir__simdf_load( o0, output3 ); stbir__simdf_madd( o0, o0, r0, stbir__simdfX_cast_to_simdf4( c3 ) ); stbir__simdf_store( output3, o0 ); )
12223 stbIF4( stbir__simdf_load( o0, output4 ); stbir__simdf_madd( o0, o0, r0, stbir__simdfX_cast_to_simdf4( c4 ) ); stbir__simdf_store( output4, o0 ); )
12224 stbIF5( stbir__simdf_load( o0, output5 ); stbir__simdf_madd( o0, o0, r0, stbir__simdfX_cast_to_simdf4( c5 ) ); stbir__simdf_store( output5, o0 ); )
12225 stbIF6( stbir__simdf_load( o0, output6 ); stbir__simdf_madd( o0, o0, r0, stbir__simdfX_cast_to_simdf4( c6 ) ); stbir__simdf_store( output6, o0 ); )
12226 stbIF7( stbir__simdf_load( o0, output7 ); stbir__simdf_madd( o0, o0, r0, stbir__simdfX_cast_to_simdf4( c7 ) ); stbir__simdf_store( output7, o0 ); )
12227 #else
12228 stbIF0( stbir__simdf_mult( o0, r0, stbir__simdfX_cast_to_simdf4( c0 ) ); stbir__simdf_store( output0, o0 ); )
12229 stbIF1( stbir__simdf_mult( o0, r0, stbir__simdfX_cast_to_simdf4( c1 ) ); stbir__simdf_store( output1, o0 ); )
12230 stbIF2( stbir__simdf_mult( o0, r0, stbir__simdfX_cast_to_simdf4( c2 ) ); stbir__simdf_store( output2, o0 ); )
12231 stbIF3( stbir__simdf_mult( o0, r0, stbir__simdfX_cast_to_simdf4( c3 ) ); stbir__simdf_store( output3, o0 ); )
12232 stbIF4( stbir__simdf_mult( o0, r0, stbir__simdfX_cast_to_simdf4( c4 ) ); stbir__simdf_store( output4, o0 ); )
12233 stbIF5( stbir__simdf_mult( o0, r0, stbir__simdfX_cast_to_simdf4( c5 ) ); stbir__simdf_store( output5, o0 ); )
12234 stbIF6( stbir__simdf_mult( o0, r0, stbir__simdfX_cast_to_simdf4( c6 ) ); stbir__simdf_store( output6, o0 ); )
12235 stbIF7( stbir__simdf_mult( o0, r0, stbir__simdfX_cast_to_simdf4( c7 ) ); stbir__simdf_store( output7, o0 ); )
12236 #endif
12237
12238 input += 4;
12239 stbIF0( output0 += 4; )
12240 stbIF1( output1 += 4; )
12241 stbIF2( output2 += 4; )
12242 stbIF3( output3 += 4; )
12243 stbIF4( output4 += 4; )
12244 stbIF5( output5 += 4; )
12245 stbIF6( output6 += 4; )
12246 stbIF7( output7 += 4; )
12247 }
12248 }
12249 #else
12250 STBIR_NO_UNROLL_LOOP_START
12251 while ( ( (char*)input_end - (char*) input ) >= 16 )
12252 {
12253 float r0, r1, r2, r3;
12254 STBIR_NO_UNROLL(input);
12255
12256 r0 = input[0], r1 = input[1], r2 = input[2], r3 = input[3];
12257
12258 #ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
12259 stbIF0( output0[0] += ( r0 * c0s ); output0[1] += ( r1 * c0s ); output0[2] += ( r2 * c0s ); output0[3] += ( r3 * c0s ); )
12260 stbIF1( output1[0] += ( r0 * c1s ); output1[1] += ( r1 * c1s ); output1[2] += ( r2 * c1s ); output1[3] += ( r3 * c1s ); )
12261 stbIF2( output2[0] += ( r0 * c2s ); output2[1] += ( r1 * c2s ); output2[2] += ( r2 * c2s ); output2[3] += ( r3 * c2s ); )
12262 stbIF3( output3[0] += ( r0 * c3s ); output3[1] += ( r1 * c3s ); output3[2] += ( r2 * c3s ); output3[3] += ( r3 * c3s ); )
12263 stbIF4( output4[0] += ( r0 * c4s ); output4[1] += ( r1 * c4s ); output4[2] += ( r2 * c4s ); output4[3] += ( r3 * c4s ); )
12264 stbIF5( output5[0] += ( r0 * c5s ); output5[1] += ( r1 * c5s ); output5[2] += ( r2 * c5s ); output5[3] += ( r3 * c5s ); )
12265 stbIF6( output6[0] += ( r0 * c6s ); output6[1] += ( r1 * c6s ); output6[2] += ( r2 * c6s ); output6[3] += ( r3 * c6s ); )
12266 stbIF7( output7[0] += ( r0 * c7s ); output7[1] += ( r1 * c7s ); output7[2] += ( r2 * c7s ); output7[3] += ( r3 * c7s ); )
12267 #else
12268 stbIF0( output0[0] = ( r0 * c0s ); output0[1] = ( r1 * c0s ); output0[2] = ( r2 * c0s ); output0[3] = ( r3 * c0s ); )
12269 stbIF1( output1[0] = ( r0 * c1s ); output1[1] = ( r1 * c1s ); output1[2] = ( r2 * c1s ); output1[3] = ( r3 * c1s ); )
12270 stbIF2( output2[0] = ( r0 * c2s ); output2[1] = ( r1 * c2s ); output2[2] = ( r2 * c2s ); output2[3] = ( r3 * c2s ); )
12271 stbIF3( output3[0] = ( r0 * c3s ); output3[1] = ( r1 * c3s ); output3[2] = ( r2 * c3s ); output3[3] = ( r3 * c3s ); )
12272 stbIF4( output4[0] = ( r0 * c4s ); output4[1] = ( r1 * c4s ); output4[2] = ( r2 * c4s ); output4[3] = ( r3 * c4s ); )
12273 stbIF5( output5[0] = ( r0 * c5s ); output5[1] = ( r1 * c5s ); output5[2] = ( r2 * c5s ); output5[3] = ( r3 * c5s ); )
12274 stbIF6( output6[0] = ( r0 * c6s ); output6[1] = ( r1 * c6s ); output6[2] = ( r2 * c6s ); output6[3] = ( r3 * c6s ); )
12275 stbIF7( output7[0] = ( r0 * c7s ); output7[1] = ( r1 * c7s ); output7[2] = ( r2 * c7s ); output7[3] = ( r3 * c7s ); )
12276 #endif
12277
12278 input += 4;
12279 stbIF0( output0 += 4; )
12280 stbIF1( output1 += 4; )
12281 stbIF2( output2 += 4; )
12282 stbIF3( output3 += 4; )
12283 stbIF4( output4 += 4; )
12284 stbIF5( output5 += 4; )
12285 stbIF6( output6 += 4; )
12286 stbIF7( output7 += 4; )
12287 }
12288 #endif
12289 STBIR_NO_UNROLL_LOOP_START
12290 while ( input < input_end )
12291 {
12292 float r = input[0];
12293 STBIR_NO_UNROLL(output0);
12294
12295 #ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
12296 stbIF0( output0[0] += ( r * c0s ); )
12297 stbIF1( output1[0] += ( r * c1s ); )
12298 stbIF2( output2[0] += ( r * c2s ); )
12299 stbIF3( output3[0] += ( r * c3s ); )
12300 stbIF4( output4[0] += ( r * c4s ); )
12301 stbIF5( output5[0] += ( r * c5s ); )
12302 stbIF6( output6[0] += ( r * c6s ); )
12303 stbIF7( output7[0] += ( r * c7s ); )
12304 #else
12305 stbIF0( output0[0] = ( r * c0s ); )
12306 stbIF1( output1[0] = ( r * c1s ); )
12307 stbIF2( output2[0] = ( r * c2s ); )
12308 stbIF3( output3[0] = ( r * c3s ); )
12309 stbIF4( output4[0] = ( r * c4s ); )
12310 stbIF5( output5[0] = ( r * c5s ); )
12311 stbIF6( output6[0] = ( r * c6s ); )
12312 stbIF7( output7[0] = ( r * c7s ); )
12313 #endif
12314
12315 ++input;
12316 stbIF0( ++output0; )
12317 stbIF1( ++output1; )
12318 stbIF2( ++output2; )
12319 stbIF3( ++output3; )
12320 stbIF4( ++output4; )
12321 stbIF5( ++output5; )
12322 stbIF6( ++output6; )
12323 stbIF7( ++output7; )
12324 }
12325}
12326
12327static void STBIR_chans( stbir__vertical_gather_with_,_coeffs)( float * outputp, float const * vertical_coefficients, float const ** inputs, float const * input0_end )
12328{
12329 float STBIR_SIMD_STREAMOUT_PTR( * ) output = outputp;
12330
12331 stbIF0( float const * input0 = inputs[0]; float c0s = vertical_coefficients[0]; )
12332 stbIF1( float const * input1 = inputs[1]; float c1s = vertical_coefficients[1]; )
12333 stbIF2( float const * input2 = inputs[2]; float c2s = vertical_coefficients[2]; )
12334 stbIF3( float const * input3 = inputs[3]; float c3s = vertical_coefficients[3]; )
12335 stbIF4( float const * input4 = inputs[4]; float c4s = vertical_coefficients[4]; )
12336 stbIF5( float const * input5 = inputs[5]; float c5s = vertical_coefficients[5]; )
12337 stbIF6( float const * input6 = inputs[6]; float c6s = vertical_coefficients[6]; )
12338 stbIF7( float const * input7 = inputs[7]; float c7s = vertical_coefficients[7]; )
12339
12340#if ( STBIR__vertical_channels == 1 ) && !defined(STB_IMAGE_RESIZE_VERTICAL_CONTINUE)
12341 // one coefficient within a millionth of 1 (any channel count, the single coefficient instantiation): a copy
12342 if ( ( c0s >= (1.0f-0.000001f) ) && ( c0s <= (1.0f+0.000001f) ) )
12343 {
12344 STBIR_MEMCPY( output, input0, (char*)input0_end - (char*)input0 );
12345 return;
12346 }
12347#endif
12348
12349 #ifdef STBIR_SIMD
12350 {
12351 stbIF0(stbir__simdfX c0 = stbir__simdf_frepX( c0s ); )
12352 stbIF1(stbir__simdfX c1 = stbir__simdf_frepX( c1s ); )
12353 stbIF2(stbir__simdfX c2 = stbir__simdf_frepX( c2s ); )
12354 stbIF3(stbir__simdfX c3 = stbir__simdf_frepX( c3s ); )
12355 stbIF4(stbir__simdfX c4 = stbir__simdf_frepX( c4s ); )
12356 stbIF5(stbir__simdfX c5 = stbir__simdf_frepX( c5s ); )
12357 stbIF6(stbir__simdfX c6 = stbir__simdf_frepX( c6s ); )
12358 stbIF7(stbir__simdfX c7 = stbir__simdf_frepX( c7s ); )
12359
12360 STBIR_SIMD_NO_UNROLL_LOOP_START
12361 while ( ( (char*)input0_end - (char*) input0 ) >= (16*stbir__simdfX_float_count) )
12362 {
12363 stbir__simdfX o0, o1, o2, o3, r0, r1, r2, r3;
12364 STBIR_SIMD_NO_UNROLL(output);
12365
12366 // prefetch four loop iterations ahead (doesn't affect much for small resizes, but helps with big ones)
12367 stbIF0( stbir__prefetch( input0 + (16*stbir__simdfX_float_count) ); )
12368 stbIF1( stbir__prefetch( input1 + (16*stbir__simdfX_float_count) ); )
12369 stbIF2( stbir__prefetch( input2 + (16*stbir__simdfX_float_count) ); )
12370 stbIF3( stbir__prefetch( input3 + (16*stbir__simdfX_float_count) ); )
12371 stbIF4( stbir__prefetch( input4 + (16*stbir__simdfX_float_count) ); )
12372 stbIF5( stbir__prefetch( input5 + (16*stbir__simdfX_float_count) ); )
12373 stbIF6( stbir__prefetch( input6 + (16*stbir__simdfX_float_count) ); )
12374 stbIF7( stbir__prefetch( input7 + (16*stbir__simdfX_float_count) ); )
12375
12376 #ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
12377 stbIF0( stbir__simdfX_load( o0, output ); stbir__simdfX_load( o1, output+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( o3, output+(3*stbir__simdfX_float_count) );
12378 stbir__simdfX_load( r0, input0 ); stbir__simdfX_load( r1, input0+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input0+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input0+(3*stbir__simdfX_float_count) );
12379 stbir__simdfX_madd( o0, o0, r0, c0 ); stbir__simdfX_madd( o1, o1, r1, c0 ); stbir__simdfX_madd( o2, o2, r2, c0 ); stbir__simdfX_madd( o3, o3, r3, c0 ); )
12380 #else
12381 stbIF0( stbir__simdfX_load( r0, input0 ); stbir__simdfX_load( r1, input0+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input0+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input0+(3*stbir__simdfX_float_count) );
12382 stbir__simdfX_mult( o0, r0, c0 ); stbir__simdfX_mult( o1, r1, c0 ); stbir__simdfX_mult( o2, r2, c0 ); stbir__simdfX_mult( o3, r3, c0 ); )
12383 #endif
12384
12385 stbIF1( stbir__simdfX_load( r0, input1 ); stbir__simdfX_load( r1, input1+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input1+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input1+(3*stbir__simdfX_float_count) );
12386 stbir__simdfX_madd( o0, o0, r0, c1 ); stbir__simdfX_madd( o1, o1, r1, c1 ); stbir__simdfX_madd( o2, o2, r2, c1 ); stbir__simdfX_madd( o3, o3, r3, c1 ); )
12387 stbIF2( stbir__simdfX_load( r0, input2 ); stbir__simdfX_load( r1, input2+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input2+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input2+(3*stbir__simdfX_float_count) );
12388 stbir__simdfX_madd( o0, o0, r0, c2 ); stbir__simdfX_madd( o1, o1, r1, c2 ); stbir__simdfX_madd( o2, o2, r2, c2 ); stbir__simdfX_madd( o3, o3, r3, c2 ); )
12389 stbIF3( stbir__simdfX_load( r0, input3 ); stbir__simdfX_load( r1, input3+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input3+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input3+(3*stbir__simdfX_float_count) );
12390 stbir__simdfX_madd( o0, o0, r0, c3 ); stbir__simdfX_madd( o1, o1, r1, c3 ); stbir__simdfX_madd( o2, o2, r2, c3 ); stbir__simdfX_madd( o3, o3, r3, c3 ); )
12391 stbIF4( stbir__simdfX_load( r0, input4 ); stbir__simdfX_load( r1, input4+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input4+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input4+(3*stbir__simdfX_float_count) );
12392 stbir__simdfX_madd( o0, o0, r0, c4 ); stbir__simdfX_madd( o1, o1, r1, c4 ); stbir__simdfX_madd( o2, o2, r2, c4 ); stbir__simdfX_madd( o3, o3, r3, c4 ); )
12393 stbIF5( stbir__simdfX_load( r0, input5 ); stbir__simdfX_load( r1, input5+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input5+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input5+(3*stbir__simdfX_float_count) );
12394 stbir__simdfX_madd( o0, o0, r0, c5 ); stbir__simdfX_madd( o1, o1, r1, c5 ); stbir__simdfX_madd( o2, o2, r2, c5 ); stbir__simdfX_madd( o3, o3, r3, c5 ); )
12395 stbIF6( stbir__simdfX_load( r0, input6 ); stbir__simdfX_load( r1, input6+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input6+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input6+(3*stbir__simdfX_float_count) );
12396 stbir__simdfX_madd( o0, o0, r0, c6 ); stbir__simdfX_madd( o1, o1, r1, c6 ); stbir__simdfX_madd( o2, o2, r2, c6 ); stbir__simdfX_madd( o3, o3, r3, c6 ); )
12397 stbIF7( stbir__simdfX_load( r0, input7 ); stbir__simdfX_load( r1, input7+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input7+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input7+(3*stbir__simdfX_float_count) );
12398 stbir__simdfX_madd( o0, o0, r0, c7 ); stbir__simdfX_madd( o1, o1, r1, c7 ); stbir__simdfX_madd( o2, o2, r2, c7 ); stbir__simdfX_madd( o3, o3, r3, c7 ); )
12399
12400 stbir__simdfX_store( output, o0 ); stbir__simdfX_store( output+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output+(3*stbir__simdfX_float_count), o3 );
12401 output += (4*stbir__simdfX_float_count);
12402 stbIF0( input0 += (4*stbir__simdfX_float_count); )
12403 stbIF1( input1 += (4*stbir__simdfX_float_count); )
12404 stbIF2( input2 += (4*stbir__simdfX_float_count); )
12405 stbIF3( input3 += (4*stbir__simdfX_float_count); )
12406 stbIF4( input4 += (4*stbir__simdfX_float_count); )
12407 stbIF5( input5 += (4*stbir__simdfX_float_count); )
12408 stbIF6( input6 += (4*stbir__simdfX_float_count); )
12409 stbIF7( input7 += (4*stbir__simdfX_float_count); )
12410 }
12411
12412 STBIR_SIMD_NO_UNROLL_LOOP_START
12413 while ( ( (char*)input0_end - (char*) input0 ) >= 16 )
12414 {
12415 stbir__simdf o0, r0;
12416 STBIR_SIMD_NO_UNROLL(output);
12417
12418 #ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
12419 stbIF0( stbir__simdf_load( o0, output ); stbir__simdf_load( r0, input0 ); stbir__simdf_madd( o0, o0, r0, stbir__simdfX_cast_to_simdf4( c0 ) ); )
12420 #else
12421 stbIF0( stbir__simdf_load( r0, input0 ); stbir__simdf_mult( o0, r0, stbir__simdfX_cast_to_simdf4( c0 ) ); )
12422 #endif
12423 stbIF1( stbir__simdf_load( r0, input1 ); stbir__simdf_madd( o0, o0, r0, stbir__simdfX_cast_to_simdf4( c1 ) ); )
12424 stbIF2( stbir__simdf_load( r0, input2 ); stbir__simdf_madd( o0, o0, r0, stbir__simdfX_cast_to_simdf4( c2 ) ); )
12425 stbIF3( stbir__simdf_load( r0, input3 ); stbir__simdf_madd( o0, o0, r0, stbir__simdfX_cast_to_simdf4( c3 ) ); )
12426 stbIF4( stbir__simdf_load( r0, input4 ); stbir__simdf_madd( o0, o0, r0, stbir__simdfX_cast_to_simdf4( c4 ) ); )
12427 stbIF5( stbir__simdf_load( r0, input5 ); stbir__simdf_madd( o0, o0, r0, stbir__simdfX_cast_to_simdf4( c5 ) ); )
12428 stbIF6( stbir__simdf_load( r0, input6 ); stbir__simdf_madd( o0, o0, r0, stbir__simdfX_cast_to_simdf4( c6 ) ); )
12429 stbIF7( stbir__simdf_load( r0, input7 ); stbir__simdf_madd( o0, o0, r0, stbir__simdfX_cast_to_simdf4( c7 ) ); )
12430
12431 stbir__simdf_store( output, o0 );
12432 output += 4;
12433 stbIF0( input0 += 4; )
12434 stbIF1( input1 += 4; )
12435 stbIF2( input2 += 4; )
12436 stbIF3( input3 += 4; )
12437 stbIF4( input4 += 4; )
12438 stbIF5( input5 += 4; )
12439 stbIF6( input6 += 4; )
12440 stbIF7( input7 += 4; )
12441 }
12442 }
12443 #else
12444 STBIR_NO_UNROLL_LOOP_START
12445 while ( ( (char*)input0_end - (char*) input0 ) >= 16 )
12446 {
12447 float o0, o1, o2, o3;
12448 STBIR_NO_UNROLL(output);
12449 #ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
12450 stbIF0( o0 = output[0] + input0[0] * c0s; o1 = output[1] + input0[1] * c0s; o2 = output[2] + input0[2] * c0s; o3 = output[3] + input0[3] * c0s; )
12451 #else
12452 stbIF0( o0 = input0[0] * c0s; o1 = input0[1] * c0s; o2 = input0[2] * c0s; o3 = input0[3] * c0s; )
12453 #endif
12454 stbIF1( o0 += input1[0] * c1s; o1 += input1[1] * c1s; o2 += input1[2] * c1s; o3 += input1[3] * c1s; )
12455 stbIF2( o0 += input2[0] * c2s; o1 += input2[1] * c2s; o2 += input2[2] * c2s; o3 += input2[3] * c2s; )
12456 stbIF3( o0 += input3[0] * c3s; o1 += input3[1] * c3s; o2 += input3[2] * c3s; o3 += input3[3] * c3s; )
12457 stbIF4( o0 += input4[0] * c4s; o1 += input4[1] * c4s; o2 += input4[2] * c4s; o3 += input4[3] * c4s; )
12458 stbIF5( o0 += input5[0] * c5s; o1 += input5[1] * c5s; o2 += input5[2] * c5s; o3 += input5[3] * c5s; )
12459 stbIF6( o0 += input6[0] * c6s; o1 += input6[1] * c6s; o2 += input6[2] * c6s; o3 += input6[3] * c6s; )
12460 stbIF7( o0 += input7[0] * c7s; o1 += input7[1] * c7s; o2 += input7[2] * c7s; o3 += input7[3] * c7s; )
12461 output[0] = o0; output[1] = o1; output[2] = o2; output[3] = o3;
12462 output += 4;
12463 stbIF0( input0 += 4; )
12464 stbIF1( input1 += 4; )
12465 stbIF2( input2 += 4; )
12466 stbIF3( input3 += 4; )
12467 stbIF4( input4 += 4; )
12468 stbIF5( input5 += 4; )
12469 stbIF6( input6 += 4; )
12470 stbIF7( input7 += 4; )
12471 }
12472 #endif
12473 STBIR_NO_UNROLL_LOOP_START
12474 while ( input0 < input0_end )
12475 {
12476 float o0;
12477 STBIR_NO_UNROLL(output);
12478 #ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
12479 stbIF0( o0 = output[0] + input0[0] * c0s; )
12480 #else
12481 stbIF0( o0 = input0[0] * c0s; )
12482 #endif
12483 stbIF1( o0 += input1[0] * c1s; )
12484 stbIF2( o0 += input2[0] * c2s; )
12485 stbIF3( o0 += input3[0] * c3s; )
12486 stbIF4( o0 += input4[0] * c4s; )
12487 stbIF5( o0 += input5[0] * c5s; )
12488 stbIF6( o0 += input6[0] * c6s; )
12489 stbIF7( o0 += input7[0] * c7s; )
12490 output[0] = o0;
12491 ++output;
12492 stbIF0( ++input0; )
12493 stbIF1( ++input1; )
12494 stbIF2( ++input2; )
12495 stbIF3( ++input3; )
12496 stbIF4( ++input4; )
12497 stbIF5( ++input5; )
12498 stbIF6( ++input6; )
12499 stbIF7( ++input7; )
12500 }
12501}
12502
12503#undef stbIF0
12504#undef stbIF1
12505#undef stbIF2
12506#undef stbIF3
12507#undef stbIF4
12508#undef stbIF5
12509#undef stbIF6
12510#undef stbIF7
12511#undef STB_IMAGE_RESIZE_DO_VERTICALS
12512#undef STBIR__vertical_channels
12513#undef STB_IMAGE_RESIZE_DO_HORIZONTALS
12514#undef STBIR_strs_join24
12515#undef STBIR_strs_join14
12516#undef STBIR_chans
12517#ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
12518#undef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
12519#endif
12520
12521#else // !STB_IMAGE_RESIZE_DO_VERTICALS
12522
12523#define STBIR_chans( start, end ) STBIR_strs_join1(start,STBIR__horizontal_channels,end)
12524
12525#ifndef stbir__2_coeff_only
12526#define stbir__2_coeff_only() \
12527 stbir__1_coeff_only(); \
12528 stbir__1_coeff_remnant(1);
12529#endif
12530
12531#ifndef stbir__2_coeff_remnant
12532#define stbir__2_coeff_remnant( ofs ) \
12533 stbir__1_coeff_remnant(ofs); \
12534 stbir__1_coeff_remnant((ofs)+1);
12535#endif
12536
12537#ifndef stbir__3_coeff_only
12538#define stbir__3_coeff_only() \
12539 stbir__2_coeff_only(); \
12540 stbir__1_coeff_remnant(2);
12541#endif
12542
12543#ifndef stbir__3_coeff_remnant
12544#define stbir__3_coeff_remnant( ofs ) \
12545 stbir__2_coeff_remnant(ofs); \
12546 stbir__1_coeff_remnant((ofs)+2);
12547#endif
12548
12549#ifndef stbir__3_coeff_setup
12550#define stbir__3_coeff_setup()
12551#endif
12552
12553#ifndef stbir__4_coeff_start
12554#define stbir__4_coeff_start() \
12555 stbir__2_coeff_only(); \
12556 stbir__2_coeff_remnant(2);
12557#endif
12558
12559#ifndef stbir__4_coeff_continue_from_4
12560#define stbir__4_coeff_continue_from_4( ofs ) \
12561 stbir__2_coeff_remnant(ofs); \
12562 stbir__2_coeff_remnant((ofs)+2);
12563#endif
12564
12565#ifndef stbir__store_output_tiny
12566#define stbir__store_output_tiny stbir__store_output
12567#endif
12568
12569static void STBIR_chans( stbir__horizontal_gather_,_channels_with_1_coeff)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
12570{
12571 float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
12572 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
12573 STBIR_SIMD_NO_UNROLL_LOOP_START
12574 do {
12575 float const * decode = decode_buffer;
12576 float const * hc = column->coefficients;
12577 stbir__1_coeff_only();
12578 stbir__store_output_tiny();
12579 } while ( output < output_end );
12580}
12581
12582static void STBIR_chans( stbir__horizontal_gather_,_channels_with_2_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
12583{
12584 float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
12585 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
12586 #ifdef stbir__x4_start
12587 // four outputs at a time while they last (one channel only, see the stbir__x4 macros)
12588 while ( ( output_end - output ) >= 4 )
12589 {
12590 stbir__x4_setup();
12591 stbir__x4_2taps();
12592 stbir__x4_store();
12593 }
12594 if ( output < output_end )
12595 #endif
12596 STBIR_SIMD_NO_UNROLL_LOOP_START
12597 do {
12598 float const * decode = decode_buffer;
12599 float const * hc = column->coefficients;
12600 stbir__2_coeff_only();
12601 stbir__store_output_tiny();
12602 } while ( output < output_end );
12603}
12604
12605static void STBIR_chans( stbir__horizontal_gather_,_channels_with_3_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
12606{
12607 float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
12608 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
12609 #ifdef stbir__x4_start
12610 // four outputs at a time while they last (one channel only, see the stbir__x4 macros)
12611 while ( ( output_end - output ) >= 4 )
12612 {
12613 stbir__x4_setup();
12614 stbir__x4_start();
12615 stbir__x4_finish3();
12616 stbir__x4_store();
12617 }
12618 if ( output < output_end )
12619 #endif
12620 STBIR_SIMD_NO_UNROLL_LOOP_START
12621 do {
12622 float const * decode = decode_buffer;
12623 float const * hc = column->coefficients;
12624 stbir__3_coeff_only();
12625 stbir__store_output_tiny();
12626 } while ( output < output_end );
12627}
12628
12629static void STBIR_chans( stbir__horizontal_gather_,_channels_with_4_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
12630{
12631 float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
12632 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
12633 #ifdef stbir__x4_start
12634 // four outputs at a time while they last (one channel only, see the stbir__x4 macros)
12635 while ( ( output_end - output ) >= 4 )
12636 {
12637 stbir__x4_setup();
12638 stbir__x4_start();
12639 stbir__x4_finish();
12640 stbir__x4_store();
12641 }
12642 if ( output < output_end )
12643 #endif
12644 STBIR_SIMD_NO_UNROLL_LOOP_START
12645 do {
12646 float const * decode = decode_buffer;
12647 float const * hc = column->coefficients;
12648 stbir__4_coeff_start();
12649 stbir__store_output();
12650 } while ( output < output_end );
12651}
12652
12653static void STBIR_chans( stbir__horizontal_gather_,_channels_with_5_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
12654{
12655 float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
12656 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
12657 #ifdef stbir__x4_start
12658 // four outputs at a time while they last (one channel only, see the stbir__x4 macros)
12659 while ( ( output_end - output ) >= 4 )
12660 {
12661 stbir__x4_setup();
12662 stbir__x4_start();
12663 stbir__x4_rem1(4);
12664 stbir__x4_finish();
12665 stbir__x4_store();
12666 }
12667 if ( output < output_end )
12668 #endif
12669 STBIR_SIMD_NO_UNROLL_LOOP_START
12670 do {
12671 float const * decode = decode_buffer;
12672 float const * hc = column->coefficients;
12673 stbir__4_coeff_start();
12674 stbir__1_coeff_remnant(4);
12675 stbir__store_output();
12676 } while ( output < output_end );
12677}
12678
12679static void STBIR_chans( stbir__horizontal_gather_,_channels_with_6_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
12680{
12681 float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
12682 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
12683 #ifdef stbir__x4_start
12684 // four outputs at a time while they last (one channel only, see the stbir__x4 macros)
12685 while ( ( output_end - output ) >= 4 )
12686 {
12687 stbir__x4_setup();
12688 stbir__x4_start();
12689 stbir__x4_rem2(4);
12690 stbir__x4_finish();
12691 stbir__x4_store();
12692 }
12693 if ( output < output_end )
12694 #endif
12695 STBIR_SIMD_NO_UNROLL_LOOP_START
12696 do {
12697 float const * decode = decode_buffer;
12698 float const * hc = column->coefficients;
12699 stbir__4_coeff_start();
12700 stbir__2_coeff_remnant(4);
12701 stbir__store_output();
12702 } while ( output < output_end );
12703}
12704
12705static void STBIR_chans( stbir__horizontal_gather_,_channels_with_7_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
12706{
12707 float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
12708 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
12709 stbir__3_coeff_setup();
12710 #ifdef stbir__x4_start
12711 // four outputs at a time while they last (one channel only, see the stbir__x4 macros)
12712 while ( ( output_end - output ) >= 4 )
12713 {
12714 stbir__x4_setup();
12715 stbir__x4_start();
12716 stbir__x4_rem3(4);
12717 stbir__x4_finish();
12718 stbir__x4_store();
12719 }
12720 if ( output < output_end )
12721 #endif
12722 STBIR_SIMD_NO_UNROLL_LOOP_START
12723 do {
12724 float const * decode = decode_buffer;
12725 float const * hc = column->coefficients;
12726
12727 stbir__4_coeff_start();
12728 stbir__3_coeff_remnant(4);
12729 stbir__store_output();
12730 } while ( output < output_end );
12731}
12732
12733static void STBIR_chans( stbir__horizontal_gather_,_channels_with_8_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
12734{
12735 float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
12736 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
12737 #ifdef stbir__x4_start
12738 // four outputs at a time while they last (one channel only, see the stbir__x4 macros)
12739 while ( ( output_end - output ) >= 4 )
12740 {
12741 stbir__x4_setup();
12742 stbir__x4_start();
12743 stbir__x4_cont(4);
12744 stbir__x4_finish();
12745 stbir__x4_store();
12746 }
12747 if ( output < output_end )
12748 #endif
12749 STBIR_SIMD_NO_UNROLL_LOOP_START
12750 do {
12751 float const * decode = decode_buffer;
12752 float const * hc = column->coefficients;
12753 stbir__4_coeff_start();
12754 stbir__4_coeff_continue_from_4(4);
12755 stbir__store_output();
12756 } while ( output < output_end );
12757}
12758
12759static void STBIR_chans( stbir__horizontal_gather_,_channels_with_9_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
12760{
12761 float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
12762 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
12763 #ifdef stbir__x4_start
12764 // four outputs at a time while they last (one channel only, see the stbir__x4 macros)
12765 while ( ( output_end - output ) >= 4 )
12766 {
12767 stbir__x4_setup();
12768 stbir__x4_start();
12769 stbir__x4_cont(4);
12770 stbir__x4_rem1(8);
12771 stbir__x4_finish();
12772 stbir__x4_store();
12773 }
12774 if ( output < output_end )
12775 #endif
12776 STBIR_SIMD_NO_UNROLL_LOOP_START
12777 do {
12778 float const * decode = decode_buffer;
12779 float const * hc = column->coefficients;
12780 stbir__4_coeff_start();
12781 stbir__4_coeff_continue_from_4(4);
12782 stbir__1_coeff_remnant(8);
12783 stbir__store_output();
12784 } while ( output < output_end );
12785}
12786
12787static void STBIR_chans( stbir__horizontal_gather_,_channels_with_10_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
12788{
12789 float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
12790 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
12791 #ifdef stbir__x4_start
12792 // four outputs at a time while they last (one channel only, see the stbir__x4 macros)
12793 while ( ( output_end - output ) >= 4 )
12794 {
12795 stbir__x4_setup();
12796 stbir__x4_start();
12797 stbir__x4_cont(4);
12798 stbir__x4_rem2(8);
12799 stbir__x4_finish();
12800 stbir__x4_store();
12801 }
12802 if ( output < output_end )
12803 #endif
12804 STBIR_SIMD_NO_UNROLL_LOOP_START
12805 do {
12806 float const * decode = decode_buffer;
12807 float const * hc = column->coefficients;
12808 stbir__4_coeff_start();
12809 stbir__4_coeff_continue_from_4(4);
12810 stbir__2_coeff_remnant(8);
12811 stbir__store_output();
12812 } while ( output < output_end );
12813}
12814
12815static void STBIR_chans( stbir__horizontal_gather_,_channels_with_11_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
12816{
12817 float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
12818 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
12819 stbir__3_coeff_setup();
12820 #ifdef stbir__x4_start
12821 // four outputs at a time while they last (one channel only, see the stbir__x4 macros)
12822 while ( ( output_end - output ) >= 4 )
12823 {
12824 stbir__x4_setup();
12825 stbir__x4_start();
12826 stbir__x4_cont(4);
12827 stbir__x4_rem3(8);
12828 stbir__x4_finish();
12829 stbir__x4_store();
12830 }
12831 if ( output < output_end )
12832 #endif
12833 STBIR_SIMD_NO_UNROLL_LOOP_START
12834 do {
12835 float const * decode = decode_buffer;
12836 float const * hc = column->coefficients;
12837 stbir__4_coeff_start();
12838 stbir__4_coeff_continue_from_4(4);
12839 stbir__3_coeff_remnant(8);
12840 stbir__store_output();
12841 } while ( output < output_end );
12842}
12843
12844static void STBIR_chans( stbir__horizontal_gather_,_channels_with_12_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
12845{
12846 float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
12847 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
12848 #ifdef stbir__x4_start
12849 // four outputs at a time while they last (one channel only, see the stbir__x4 macros)
12850 while ( ( output_end - output ) >= 4 )
12851 {
12852 stbir__x4_setup();
12853 stbir__x4_start();
12854 stbir__x4_cont(4);
12855 stbir__x4_cont(8);
12856 stbir__x4_finish();
12857 stbir__x4_store();
12858 }
12859 if ( output < output_end )
12860 #endif
12861 STBIR_SIMD_NO_UNROLL_LOOP_START
12862 do {
12863 float const * decode = decode_buffer;
12864 float const * hc = column->coefficients;
12865 stbir__4_coeff_start();
12866 stbir__4_coeff_continue_from_4(4);
12867 stbir__4_coeff_continue_from_4(8);
12868 stbir__store_output();
12869 } while ( output < output_end );
12870}
12871
12872static void STBIR_chans( stbir__horizontal_gather_,_channels_with_13_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
12873{
12874 float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
12875 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
12876 #ifdef stbir__x4_start
12877 // four outputs at a time while they last (one channel only, see the stbir__x4 macros)
12878 while ( ( output_end - output ) >= 4 )
12879 {
12880 stbir__x4_setup();
12881 stbir__x4_start();
12882 stbir__x4_cont(4);
12883 stbir__x4_cont(8);
12884 stbir__x4_rem1(12);
12885 stbir__x4_finish();
12886 stbir__x4_store();
12887 }
12888 if ( output < output_end )
12889 #endif
12890 STBIR_SIMD_NO_UNROLL_LOOP_START
12891 do {
12892 float const * decode = decode_buffer;
12893 float const * hc = column->coefficients;
12894 stbir__4_coeff_start();
12895 stbir__4_coeff_continue_from_4(4);
12896 stbir__4_coeff_continue_from_4(8);
12897 stbir__1_coeff_remnant(12);
12898 stbir__store_output();
12899 } while ( output < output_end );
12900}
12901
12902static void STBIR_chans( stbir__horizontal_gather_,_channels_with_14_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
12903{
12904 float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
12905 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
12906 #ifdef stbir__x4_start
12907 // four outputs at a time while they last (one channel only, see the stbir__x4 macros)
12908 while ( ( output_end - output ) >= 4 )
12909 {
12910 stbir__x4_setup();
12911 stbir__x4_start();
12912 stbir__x4_cont(4);
12913 stbir__x4_cont(8);
12914 stbir__x4_rem2(12);
12915 stbir__x4_finish();
12916 stbir__x4_store();
12917 }
12918 if ( output < output_end )
12919 #endif
12920 STBIR_SIMD_NO_UNROLL_LOOP_START
12921 do {
12922 float const * decode = decode_buffer;
12923 float const * hc = column->coefficients;
12924 stbir__4_coeff_start();
12925 stbir__4_coeff_continue_from_4(4);
12926 stbir__4_coeff_continue_from_4(8);
12927 stbir__2_coeff_remnant(12);
12928 stbir__store_output();
12929 } while ( output < output_end );
12930}
12931
12932static void STBIR_chans( stbir__horizontal_gather_,_channels_with_15_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
12933{
12934 float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
12935 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
12936 stbir__3_coeff_setup();
12937 #ifdef stbir__x4_start
12938 // four outputs at a time while they last (one channel only, see the stbir__x4 macros)
12939 while ( ( output_end - output ) >= 4 )
12940 {
12941 stbir__x4_setup();
12942 stbir__x4_start();
12943 stbir__x4_cont(4);
12944 stbir__x4_cont(8);
12945 stbir__x4_rem3(12);
12946 stbir__x4_finish();
12947 stbir__x4_store();
12948 }
12949 if ( output < output_end )
12950 #endif
12951 STBIR_SIMD_NO_UNROLL_LOOP_START
12952 do {
12953 float const * decode = decode_buffer;
12954 float const * hc = column->coefficients;
12955 stbir__4_coeff_start();
12956 stbir__4_coeff_continue_from_4(4);
12957 stbir__4_coeff_continue_from_4(8);
12958 stbir__3_coeff_remnant(12);
12959 stbir__store_output();
12960 } while ( output < output_end );
12961}
12962
12963static void STBIR_chans( stbir__horizontal_gather_,_channels_with_16_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
12964{
12965 float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
12966 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
12967 #ifdef stbir__x4_start
12968 // four outputs at a time while they last (one channel only, see the stbir__x4 macros)
12969 while ( ( output_end - output ) >= 4 )
12970 {
12971 stbir__x4_setup();
12972 stbir__x4_start();
12973 stbir__x4_cont(4);
12974 stbir__x4_cont(8);
12975 stbir__x4_cont(12);
12976 stbir__x4_finish();
12977 stbir__x4_store();
12978 }
12979 if ( output < output_end )
12980 #endif
12981 STBIR_SIMD_NO_UNROLL_LOOP_START
12982 do {
12983 float const * decode = decode_buffer;
12984 float const * hc = column->coefficients;
12985 stbir__4_coeff_start();
12986 stbir__4_coeff_continue_from_4(4);
12987 stbir__4_coeff_continue_from_4(8);
12988 stbir__4_coeff_continue_from_4(12);
12989 stbir__store_output();
12990 } while ( output < output_end );
12991}
12992
12993static void STBIR_chans( stbir__horizontal_gather_,_channels_with_17_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
12994{
12995 float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
12996 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
12997 #ifdef stbir__x4_start
12998 // four outputs at a time while they last (one channel only, see the stbir__x4 macros)
12999 while ( ( output_end - output ) >= 4 )
13000 {
13001 stbir__x4_setup();
13002 stbir__x4_start();
13003 stbir__x4_cont(4);
13004 stbir__x4_cont(8);
13005 stbir__x4_cont(12);
13006 stbir__x4_rem1(16);
13007 stbir__x4_finish();
13008 stbir__x4_store();
13009 }
13010 if ( output < output_end )
13011 #endif
13012 STBIR_SIMD_NO_UNROLL_LOOP_START
13013 do {
13014 float const * decode = decode_buffer;
13015 float const * hc = column->coefficients;
13016 stbir__4_coeff_start();
13017 stbir__4_coeff_continue_from_4(4);
13018 stbir__4_coeff_continue_from_4(8);
13019 stbir__4_coeff_continue_from_4(12);
13020 stbir__1_coeff_remnant(16);
13021 stbir__store_output();
13022 } while ( output < output_end );
13023}
13024
13025static void STBIR_chans( stbir__horizontal_gather_,_channels_with_n_coeffs_mod0 )( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
13026{
13027 float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
13028 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
13029 STBIR_SIMD_NO_UNROLL_LOOP_START
13030 do {
13031 float const * decode = decode_buffer;
13032 int n = ( column->width - 4 + 3 ) >> 2;
13033 float const * hc = column->coefficients;
13034
13035 stbir__4_coeff_start();
13036 STBIR_SIMD_NO_UNROLL_LOOP_START
13037 do {
13038 hc += 4;
13039 decode += STBIR__horizontal_channels * 4;
13040 stbir__4_coeff_continue_from_4( 0 );
13041 --n;
13042 } while ( n > 0 );
13043 stbir__store_output();
13044 } while ( output < output_end );
13045}
13046
13047static void STBIR_chans( stbir__horizontal_gather_,_channels_with_n_coeffs_mod1 )( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
13048{
13049 float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
13050 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
13051 STBIR_SIMD_NO_UNROLL_LOOP_START
13052 do {
13053 float const * decode = decode_buffer;
13054 int n = ( column->width - 5 + 3 ) >> 2;
13055 float const * hc = column->coefficients;
13056
13057 stbir__4_coeff_start();
13058 STBIR_SIMD_NO_UNROLL_LOOP_START
13059 do {
13060 hc += 4;
13061 decode += STBIR__horizontal_channels * 4;
13062 stbir__4_coeff_continue_from_4( 0 );
13063 --n;
13064 } while ( n > 0 );
13065 stbir__1_coeff_remnant( 4 );
13066 stbir__store_output();
13067 } while ( output < output_end );
13068}
13069
13070static void STBIR_chans( stbir__horizontal_gather_,_channels_with_n_coeffs_mod2 )( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
13071{
13072 float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
13073 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
13074 STBIR_SIMD_NO_UNROLL_LOOP_START
13075 do {
13076 float const * decode = decode_buffer;
13077 int n = ( column->width - 6 + 3 ) >> 2;
13078 float const * hc = column->coefficients;
13079
13080 stbir__4_coeff_start();
13081 STBIR_SIMD_NO_UNROLL_LOOP_START
13082 do {
13083 hc += 4;
13084 decode += STBIR__horizontal_channels * 4;
13085 stbir__4_coeff_continue_from_4( 0 );
13086 --n;
13087 } while ( n > 0 );
13088 stbir__2_coeff_remnant( 4 );
13089
13090 stbir__store_output();
13091 } while ( output < output_end );
13092}
13093
13094static void STBIR_chans( stbir__horizontal_gather_,_channels_with_n_coeffs_mod3 )( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
13095{
13096 float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
13097 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
13098 stbir__3_coeff_setup();
13099 STBIR_SIMD_NO_UNROLL_LOOP_START
13100 do {
13101 float const * decode = decode_buffer;
13102 int n = ( column->width - 7 + 3 ) >> 2;
13103 float const * hc = column->coefficients;
13104
13105 stbir__4_coeff_start();
13106 STBIR_SIMD_NO_UNROLL_LOOP_START
13107 do {
13108 hc += 4;
13109 decode += STBIR__horizontal_channels * 4;
13110 stbir__4_coeff_continue_from_4( 0 );
13111 --n;
13112 } while ( n > 0 );
13113 stbir__3_coeff_remnant( 4 );
13114
13115 stbir__store_output();
13116 } while ( output < output_end );
13117}
13118
13119// polyphase period 1 kernels (integer downsamples, see stbir__resample_horizontal_gather): every output uses the same
13120// coefficient set, so it is loaded into registers once per run and the pixel loop is just the decode loads and
13121// multiply-adds. the same steps as the kernels above, for the widths integer downsamples produce (4, 8: 2:1 with the
13122// support 1 and 2 filters; 5, 9: the same one tap wider; 11/12/13 and 16/17: 3:1 and 4:1 with the support 2 filters -
13123// 3:1 trims to 11 because its outermost taps land exactly on the filter's zero at +-2).
13124// only for the channel counts that define the step macros.
13125#ifdef stbir__period1_setup_4
13126static void STBIR_chans( stbir__horizontal_gather_,_channels_with_4_coeffs_period1)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
13127{
13128 float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
13129 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
13130 float const * hc = column->coefficients;
13131 int advance = column->advance;
13132 stbir__period1_setup_4()
13133 STBIR__UNUSED( entry_stride );
13134 #ifdef stbir__period1_x4_start
13135 // four outputs at a time while they last (one channel only)
13136 while ( ( output_end - output ) >= 4 )
13137 {
13138 stbir__period1_x4_setup();
13139 stbir__period1_x4_start();
13140 stbir__period1_x4_store();
13141 }
13142 if ( output < output_end )
13143 #endif
13144 STBIR_SIMD_NO_UNROLL_LOOP_START
13145 do {
13146 float const * decode = decode_buffer;
13147 stbir__period1_4_coeff_start();
13148 stbir__period1_store_output();
13149 } while ( output < output_end );
13150}
13151
13152static void STBIR_chans( stbir__horizontal_gather_,_channels_with_5_coeffs_period1)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
13153{
13154 float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
13155 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
13156 float const * hc = column->coefficients;
13157 int advance = column->advance;
13158 stbir__period1_setup_5()
13159 STBIR__UNUSED( entry_stride );
13160 #ifdef stbir__period1_x4_start
13161 // four outputs at a time while they last (one channel only)
13162 while ( ( output_end - output ) >= 4 )
13163 {
13164 stbir__period1_x4_setup();
13165 stbir__period1_x4_start();
13166 stbir__period1_x4_remnant(4);
13167 stbir__period1_x4_store();
13168 }
13169 if ( output < output_end )
13170 #endif
13171 STBIR_SIMD_NO_UNROLL_LOOP_START
13172 do {
13173 float const * decode = decode_buffer;
13174 stbir__period1_4_coeff_start();
13175 stbir__period1_1_coeff_remnant(4);
13176 stbir__period1_store_output();
13177 } while ( output < output_end );
13178}
13179
13180static void STBIR_chans( stbir__horizontal_gather_,_channels_with_8_coeffs_period1)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
13181{
13182 float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
13183 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
13184 float const * hc = column->coefficients;
13185 int advance = column->advance;
13186 stbir__period1_setup_8()
13187 STBIR__UNUSED( entry_stride );
13188 #ifdef stbir__period1_x4_start
13189 // four outputs at a time while they last (one channel only)
13190 while ( ( output_end - output ) >= 4 )
13191 {
13192 stbir__period1_x4_setup();
13193 stbir__period1_x4_start();
13194 stbir__period1_x4_continue();
13195 stbir__period1_x4_store();
13196 }
13197 if ( output < output_end )
13198 #endif
13199 STBIR_SIMD_NO_UNROLL_LOOP_START
13200 do {
13201 float const * decode = decode_buffer;
13202 stbir__period1_4_coeff_start();
13203 stbir__period1_4_coeff_continue();
13204 stbir__period1_store_output();
13205 } while ( output < output_end );
13206}
13207
13208static void STBIR_chans( stbir__horizontal_gather_,_channels_with_9_coeffs_period1)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
13209{
13210 float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
13211 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
13212 float const * hc = column->coefficients;
13213 int advance = column->advance;
13214 stbir__period1_setup_9()
13215 STBIR__UNUSED( entry_stride );
13216 #ifdef stbir__period1_x4_start
13217 // four outputs at a time while they last (one channel only)
13218 while ( ( output_end - output ) >= 4 )
13219 {
13220 stbir__period1_x4_setup();
13221 stbir__period1_x4_start();
13222 stbir__period1_x4_continue();
13223 stbir__period1_x4_remnant(8);
13224 stbir__period1_x4_store();
13225 }
13226 if ( output < output_end )
13227 #endif
13228 STBIR_SIMD_NO_UNROLL_LOOP_START
13229 do {
13230 float const * decode = decode_buffer;
13231 stbir__period1_4_coeff_start();
13232 stbir__period1_4_coeff_continue();
13233 stbir__period1_1_coeff_remnant(8);
13234 stbir__period1_store_output();
13235 } while ( output < output_end );
13236}
13237
13238static void STBIR_chans( stbir__horizontal_gather_,_channels_with_11_coeffs_period1)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
13239{
13240 float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
13241 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
13242 float const * hc = column->coefficients;
13243 int advance = column->advance;
13244 stbir__period1_setup_11()
13245 STBIR__UNUSED( entry_stride );
13246 #ifdef stbir__period1_x4_start
13247 // four outputs at a time while they last (one channel only)
13248 while ( ( output_end - output ) >= 4 )
13249 {
13250 stbir__period1_x4_setup();
13251 stbir__period1_x4_start();
13252 stbir__period1_x4_continue();
13253 stbir__period1_x4_remnant3(8);
13254 stbir__period1_x4_store();
13255 }
13256 if ( output < output_end )
13257 #endif
13258 STBIR_SIMD_NO_UNROLL_LOOP_START
13259 do {
13260 float const * decode = decode_buffer;
13261 stbir__period1_4_coeff_start();
13262 stbir__period1_4_coeff_continue();
13263 stbir__period1_3_coeff_remnant(8);
13264 stbir__period1_store_output();
13265 } while ( output < output_end );
13266}
13267
13268static void STBIR_chans( stbir__horizontal_gather_,_channels_with_12_coeffs_period1)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
13269{
13270 float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
13271 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
13272 float const * hc = column->coefficients;
13273 int advance = column->advance;
13274 stbir__period1_setup_12()
13275 STBIR__UNUSED( entry_stride );
13276 #ifdef stbir__period1_x4_start
13277 // four outputs at a time while they last (one channel only)
13278 while ( ( output_end - output ) >= 4 )
13279 {
13280 stbir__period1_x4_setup();
13281 stbir__period1_x4_start();
13282 stbir__period1_x4_continue();
13283 stbir__period1_x4_continue2();
13284 stbir__period1_x4_store();
13285 }
13286 if ( output < output_end )
13287 #endif
13288 STBIR_SIMD_NO_UNROLL_LOOP_START
13289 do {
13290 float const * decode = decode_buffer;
13291 stbir__period1_4_coeff_start();
13292 stbir__period1_4_coeff_continue();
13293 stbir__period1_4_coeff_continue2();
13294 stbir__period1_store_output();
13295 } while ( output < output_end );
13296}
13297
13298static void STBIR_chans( stbir__horizontal_gather_,_channels_with_13_coeffs_period1)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
13299{
13300 float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
13301 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
13302 float const * hc = column->coefficients;
13303 int advance = column->advance;
13304 stbir__period1_setup_13()
13305 STBIR__UNUSED( entry_stride );
13306 #ifdef stbir__period1_x4_start
13307 // four outputs at a time while they last (one channel only)
13308 while ( ( output_end - output ) >= 4 )
13309 {
13310 stbir__period1_x4_setup();
13311 stbir__period1_x4_start();
13312 stbir__period1_x4_continue();
13313 stbir__period1_x4_continue2();
13314 stbir__period1_x4_remnant(12);
13315 stbir__period1_x4_store();
13316 }
13317 if ( output < output_end )
13318 #endif
13319 STBIR_SIMD_NO_UNROLL_LOOP_START
13320 do {
13321 float const * decode = decode_buffer;
13322 stbir__period1_4_coeff_start();
13323 stbir__period1_4_coeff_continue();
13324 stbir__period1_4_coeff_continue2();
13325 stbir__period1_1_coeff_remnant(12);
13326 stbir__period1_store_output();
13327 } while ( output < output_end );
13328}
13329
13330static void STBIR_chans( stbir__horizontal_gather_,_channels_with_16_coeffs_period1)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
13331{
13332 float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
13333 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
13334 float const * hc = column->coefficients;
13335 int advance = column->advance;
13336 stbir__period1_setup_16()
13337 STBIR__UNUSED( entry_stride );
13338 #ifdef stbir__period1_x4_start
13339 // four outputs at a time while they last (one channel only)
13340 while ( ( output_end - output ) >= 4 )
13341 {
13342 stbir__period1_x4_setup();
13343 stbir__period1_x4_start();
13344 stbir__period1_x4_continue();
13345 stbir__period1_x4_continue2();
13346 stbir__period1_x4_continue3();
13347 stbir__period1_x4_store();
13348 }
13349 if ( output < output_end )
13350 #endif
13351 STBIR_SIMD_NO_UNROLL_LOOP_START
13352 do {
13353 float const * decode = decode_buffer;
13354 stbir__period1_4_coeff_start();
13355 stbir__period1_4_coeff_continue();
13356 stbir__period1_4_coeff_continue2();
13357 stbir__period1_4_coeff_continue3();
13358 stbir__period1_store_output();
13359 } while ( output < output_end );
13360}
13361
13362static void STBIR_chans( stbir__horizontal_gather_,_channels_with_17_coeffs_period1)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__column_entry const * column, int entry_stride )
13363{
13364 float const * output_end = output_buffer + (ptrdiff_t)output_sub_size * STBIR__horizontal_channels;
13365 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
13366 float const * hc = column->coefficients;
13367 int advance = column->advance;
13368 stbir__period1_setup_17()
13369 STBIR__UNUSED( entry_stride );
13370 #ifdef stbir__period1_x4_start
13371 // four outputs at a time while they last (one channel only)
13372 while ( ( output_end - output ) >= 4 )
13373 {
13374 stbir__period1_x4_setup();
13375 stbir__period1_x4_start();
13376 stbir__period1_x4_continue();
13377 stbir__period1_x4_continue2();
13378 stbir__period1_x4_continue3();
13379 stbir__period1_x4_remnant(16);
13380 stbir__period1_x4_store();
13381 }
13382 if ( output < output_end )
13383 #endif
13384 STBIR_SIMD_NO_UNROLL_LOOP_START
13385 do {
13386 float const * decode = decode_buffer;
13387 stbir__period1_4_coeff_start();
13388 stbir__period1_4_coeff_continue();
13389 stbir__period1_4_coeff_continue2();
13390 stbir__period1_4_coeff_continue3();
13391 stbir__period1_1_coeff_remnant(16);
13392 stbir__period1_store_output();
13393 } while ( output < output_end );
13394}
13395
13396// by coefficient count (null = use the normal kernel over the one entry)
13397static stbir__horizontal_gather_channels_func * STBIR_chans(stbir__horizontal_gather_,_channels_period1_funcs)[18]=
13398{
13399 0, 0, 0, 0,
13400 STBIR_chans(stbir__horizontal_gather_,_channels_with_4_coeffs_period1),
13401 STBIR_chans(stbir__horizontal_gather_,_channels_with_5_coeffs_period1),
13402 0, 0,
13403 STBIR_chans(stbir__horizontal_gather_,_channels_with_8_coeffs_period1),
13404 STBIR_chans(stbir__horizontal_gather_,_channels_with_9_coeffs_period1),
13405 0,
13406 STBIR_chans(stbir__horizontal_gather_,_channels_with_11_coeffs_period1),
13407 STBIR_chans(stbir__horizontal_gather_,_channels_with_12_coeffs_period1),
13408 STBIR_chans(stbir__horizontal_gather_,_channels_with_13_coeffs_period1),
13409 0, 0,
13410 STBIR_chans(stbir__horizontal_gather_,_channels_with_16_coeffs_period1),
13411 STBIR_chans(stbir__horizontal_gather_,_channels_with_17_coeffs_period1)
13412};
13413#else
13414static stbir__horizontal_gather_channels_func * STBIR_chans(stbir__horizontal_gather_,_channels_period1_funcs)[18]= { 0 };
13415#endif
13416
13417static stbir__horizontal_gather_channels_func * STBIR_chans(stbir__horizontal_gather_,_channels_with_n_coeffs_funcs)[4]=
13418{
13419 STBIR_chans(stbir__horizontal_gather_,_channels_with_n_coeffs_mod0),
13420 STBIR_chans(stbir__horizontal_gather_,_channels_with_n_coeffs_mod1),
13421 STBIR_chans(stbir__horizontal_gather_,_channels_with_n_coeffs_mod2),
13422 STBIR_chans(stbir__horizontal_gather_,_channels_with_n_coeffs_mod3),
13423};
13424
13425static stbir__horizontal_gather_channels_func * STBIR_chans(stbir__horizontal_gather_,_channels_funcs)[17]=
13426{
13427 STBIR_chans(stbir__horizontal_gather_,_channels_with_1_coeff),
13428 STBIR_chans(stbir__horizontal_gather_,_channels_with_2_coeffs),
13429 STBIR_chans(stbir__horizontal_gather_,_channels_with_3_coeffs),
13430 STBIR_chans(stbir__horizontal_gather_,_channels_with_4_coeffs),
13431 STBIR_chans(stbir__horizontal_gather_,_channels_with_5_coeffs),
13432 STBIR_chans(stbir__horizontal_gather_,_channels_with_6_coeffs),
13433 STBIR_chans(stbir__horizontal_gather_,_channels_with_7_coeffs),
13434 STBIR_chans(stbir__horizontal_gather_,_channels_with_8_coeffs),
13435 STBIR_chans(stbir__horizontal_gather_,_channels_with_9_coeffs),
13436 STBIR_chans(stbir__horizontal_gather_,_channels_with_10_coeffs),
13437 STBIR_chans(stbir__horizontal_gather_,_channels_with_11_coeffs),
13438 STBIR_chans(stbir__horizontal_gather_,_channels_with_12_coeffs),
13439 STBIR_chans(stbir__horizontal_gather_,_channels_with_13_coeffs),
13440 STBIR_chans(stbir__horizontal_gather_,_channels_with_14_coeffs),
13441 STBIR_chans(stbir__horizontal_gather_,_channels_with_15_coeffs),
13442 STBIR_chans(stbir__horizontal_gather_,_channels_with_16_coeffs),
13443 STBIR_chans(stbir__horizontal_gather_,_channels_with_17_coeffs),
13444};
13445
13446#ifdef stbir__period1_setup_4
13447#undef stbir__period1_setup_4
13448#undef stbir__period1_setup_5
13449#undef stbir__period1_setup_8
13450#undef stbir__period1_setup_9
13451#undef stbir__period1_setup_11
13452#undef stbir__period1_setup_12
13453#undef stbir__period1_setup_13
13454#undef stbir__period1_setup_16
13455#undef stbir__period1_setup_17
13456#undef stbir__period1_4_coeff_start
13457#undef stbir__period1_4_coeff_continue
13458#undef stbir__period1_4_coeff_continue2
13459#undef stbir__period1_4_coeff_continue3
13460#undef stbir__period1_1_coeff_remnant
13461#undef stbir__period1_3_coeff_remnant
13462#undef stbir__period1_store_output
13463#endif
13464#ifdef stbir__period1_x4_start
13465#undef stbir__period1_x4_setup
13466#undef stbir__period1_x4_start
13467#undef stbir__period1_x4_next
13468#undef stbir__period1_x4_continue
13469#undef stbir__period1_x4_continue2
13470#undef stbir__period1_x4_continue3
13471#undef stbir__period1_x4_remnant
13472#undef stbir__period1_x4_remnant3
13473#undef stbir__period1_x4_store
13474#endif
13475#undef STBIR__horizontal_channels
13476#undef STB_IMAGE_RESIZE_DO_HORIZONTALS
13477#ifdef stbir__x4_start
13478#undef stbir__x4_setup
13479#undef stbir__x4_start
13480#undef stbir__x4_cont
13481#undef stbir__x4_rem1
13482#undef stbir__x4_rem2
13483#undef stbir__x4_rem3
13484#undef stbir__x4_2taps
13485#undef stbir__x4_finish
13486#undef stbir__x4_finish3
13487#undef stbir__x4_store
13488#endif
13489#undef stbir__1_coeff_only
13490#undef stbir__1_coeff_remnant
13491#undef stbir__2_coeff_only
13492#undef stbir__2_coeff_remnant
13493#undef stbir__3_coeff_only
13494#undef stbir__3_coeff_remnant
13495#undef stbir__3_coeff_setup
13496#undef stbir__4_coeff_start
13497#undef stbir__4_coeff_continue_from_4
13498#undef stbir__store_output
13499#undef stbir__store_output_tiny
13500#undef STBIR_chans
13501
13502#endif // HORIZONALS
13503
13504#undef STBIR_strs_join2
13505#undef STBIR_strs_join1
13506
13507#endif // STB_IMAGE_RESIZE_DO_HORIZONTALS/VERTICALS/CODERS
13508
13509/*
13510------------------------------------------------------------------------------
13511This software is available under 2 licenses -- choose whichever you prefer.
13512------------------------------------------------------------------------------
13513ALTERNATIVE A - MIT License
13514Copyright (c) 2017 Sean Barrett
13515Permission is hereby granted, free of charge, to any person obtaining a copy of
13516this software and associated documentation files (the "Software"), to deal in
13517the Software without restriction, including without limitation the rights to
13518use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
13519of the Software, and to permit persons to whom the Software is furnished to do
13520so, subject to the following conditions:
13521The above copyright notice and this permission notice shall be included in all
13522copies or substantial portions of the Software.
13523THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
13524IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
13525FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
13526AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
13527LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
13528OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
13529SOFTWARE.
13530------------------------------------------------------------------------------
13531ALTERNATIVE B - Public Domain (www.unlicense.org)
13532This is free and unencumbered software released into the public domain.
13533Anyone is free to copy, modify, publish, use, compile, sell, or distribute this
13534software, either in source code form or as a compiled binary, for any purpose,
13535commercial or non-commercial, and by any means.
13536In jurisdictions that recognize copyright laws, the author or authors of this
13537software dedicate any and all copyright interest in the software to the public
13538domain. We make this dedication for the benefit of the public at large and to
13539the detriment of our heirs and successors. We intend this dedication to be an
13540overt act of relinquishment in perpetuity of all present and future rights to
13541this software under copyright law.
13542THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
13543IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
13544FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
13545AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
13546ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
13547WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
13548------------------------------------------------------------------------------
13549*/
