master
stb_image_resize2.h
1/* stb_image_resize2 - v2.18 - public domain image resizing
2
3 by Jeff Roberts (v2) and Jorge L Rodriguez
4 http://github.com/nothings/stb
5
6 Can be threaded with the extended API. SSE2, AVX, Neon and WASM SIMD support. Only
7 scaling and translation is supported, no rotations or shears.
8
9 COMPILING & LINKING
10 In one C/C++ file that #includes this file, do this:
11 #define STB_IMAGE_RESIZE_IMPLEMENTATION
12 before the #include. That will create the implementation in that file.
13
14 EASY API CALLS:
15 Easy API downsamples w/Mitchell filter, upsamples w/cubic interpolation, clamps to edge.
16
17 stbir_resize_uint8_srgb( input_pixels, input_w, input_h, input_stride_in_bytes,
18 output_pixels, output_w, output_h, output_stride_in_bytes,
19 pixel_layout_enum )
20
21 stbir_resize_uint8_linear( input_pixels, input_w, input_h, input_stride_in_bytes,
22 output_pixels, output_w, output_h, output_stride_in_bytes,
23 pixel_layout_enum )
24
25 stbir_resize_float_linear( input_pixels, input_w, input_h, input_stride_in_bytes,
26 output_pixels, output_w, output_h, output_stride_in_bytes,
27 pixel_layout_enum )
28
29 If you pass NULL or zero for the output_pixels, we will allocate the output buffer
30 for you and return it from the function (free with free() or STBIR_FREE).
31 As a special case, XX_stride_in_bytes of 0 means packed continuously in memory.
32
33 API LEVELS
34 There are three levels of API - easy-to-use, medium-complexity and extended-complexity.
35
36 See the "header file" section of the source for API documentation.
37
38 ADDITIONAL DOCUMENTATION
39
40 MEMORY ALLOCATION
41 By default, we use malloc and free for memory allocation. To override the
42 memory allocation, before the implementation #include, add a:
43
44 #define STBIR_MALLOC(size,user_data) ...
45 #define STBIR_FREE(ptr,user_data) ...
46
47 Each resize makes exactly one call to malloc/free (unless you use the
48 extended API where you can do one allocation for many resizes). Under
49 address sanitizer, we do separate allocations to find overread/writes.
50
51 PERFORMANCE
52 This library was written with an emphasis on performance. When testing
53 stb_image_resize with RGBA, the fastest mode is STBIR_4CHANNEL with
54 STBIR_TYPE_UINT8 pixels and CLAMPed edges (which is what many other resize
55 libs do by default). Also, make sure SIMD is turned on of course (default
56 for 64-bit targets). Avoid WRAP edge mode if you want the fastest speed.
57
58 This library also comes with profiling built-in. If you define STBIR_PROFILE,
59 you can use the advanced API and get low-level profiling information by
60 calling stbir_resize_extended_profile_info() or stbir_resize_split_profile_info()
61 after a resize.
62
63 SIMD
64 Most of the routines have optimized SSE2, AVX, NEON and WASM versions.
65
66 On Microsoft compilers, we automatically turn on SIMD for 64-bit x64 and
67 ARM; for 32-bit x86 and ARM, you select SIMD mode by defining STBIR_SSE2 or
68 STBIR_NEON. For AVX and AVX2, we auto-select it by detecting the /arch:AVX
69 or /arch:AVX2 switches. You can also always manually turn SSE2, AVX or AVX2
70 support on by defining STBIR_SSE2, STBIR_AVX or STBIR_AVX2.
71
72 On Linux, SSE2 and Neon is on by default for 64-bit x64 or ARM64. For 32-bit,
73 we select x86 SIMD mode by whether you have -msse2, -mavx or -mavx2 enabled
74 on the command line. For 32-bit ARM, you must pass -mfpu=neon-vfpv4 for both
75 clang and GCC, but GCC also requires an additional -mfp16-format=ieee to
76 automatically enable NEON.
77
78 On x86 platforms, you can also define STBIR_FP16C to turn on FP16C instructions
79 for converting back and forth to half-floats. This is autoselected when we
80 are using AVX2. Clang and GCC also require the -mf16c switch. ARM always uses
81 the built-in half float hardware NEON instructions.
82
83 You can also tell us to use multiply-add instructions with STBIR_USE_FMA.
84 Because x86 doesn't always have fma, we turn it off by default to maintain
85 determinism across all platforms. If you don't care about non-FMA determinism
86 and are willing to restrict yourself to more recent x86 CPUs (around the AVX
87 timeframe), then fma will give you around a 15% speedup.
88
89 You can force off SIMD in all cases by defining STBIR_NO_SIMD. You can turn
90 off AVX or AVX2 specifically with STBIR_NO_AVX or STBIR_NO_AVX2. AVX is 10%
91 to 40% faster, and AVX2 is generally another 12%.
92
93 ALPHA CHANNEL
94 Most of the resizing functions provide the ability to control how the alpha
95 channel of an image is processed.
96
97 When alpha represents transparency, it is important that when combining
98 colors with filtering, the pixels should not be treated equally; they
99 should use a weighted average based on their alpha values. For example,
100 if a pixel is 1% opaque bright green and another pixel is 99% opaque
101 black and you average them, the average will be 50% opaque, but the
102 unweighted average and will be a middling green color, while the weighted
103 average will be nearly black. This means the unweighted version introduced
104 green energy that didn't exist in the source image.
105
106 (If you want to know why this makes sense, you can work out the math for
107 the following: consider what happens if you alpha composite a source image
108 over a fixed color and then average the output, vs. if you average the
109 source image pixels and then composite that over the same fixed color.
110 Only the weighted average produces the same result as the ground truth
111 composite-then-average result.)
112
113 Therefore, it is in general best to "alpha weight" the pixels when applying
114 filters to them. This essentially means multiplying the colors by the alpha
115 values before combining them, and then dividing by the alpha value at the
116 end.
117
118 The computer graphics industry introduced a technique called "premultiplied
119 alpha" or "associated alpha" in which image colors are stored in image files
120 already multiplied by their alpha. This saves some math when compositing,
121 and also avoids the need to divide by the alpha at the end (which is quite
122 inefficient). However, while premultiplied alpha is common in the movie CGI
123 industry, it is not commonplace in other industries like videogames, and most
124 consumer file formats are generally expected to contain not-premultiplied
125 colors. For example, Photoshop saves PNG files "unpremultiplied", and web
126 browsers like Chrome and Firefox expect PNG images to be unpremultiplied.
127
128 Note that there are three possibilities that might describe your image
129 and resize expectation:
130
131 1. images are not premultiplied, alpha weighting is desired
132 2. images are not premultiplied, alpha weighting is not desired
133 3. images are premultiplied
134
135 Both case #2 and case #3 require the exact same math: no alpha weighting
136 should be applied or removed. Only case 1 requires extra math operations;
137 the other two cases can be handled identically.
138
139 stb_image_resize expects case #1 by default, applying alpha weighting to
140 images, expecting the input images to be unpremultiplied. This is what the
141 COLOR+ALPHA buffer types tell the resizer to do.
142
143 When you use the pixel layouts STBIR_RGBA, STBIR_BGRA, STBIR_ARGB,
144 STBIR_ABGR, STBIR_RA, or STBIR_AR you are telling us that the pixels are
145 non-premultiplied. In these cases, the resizer will alpha weight the colors
146 (effectively creating the premultiplied image), do the filtering, and then
147 convert back to non-premult on exit.
148
149 When you use the pixel layouts STBIR_RGBA_PM, STBIR_BGRA_PM, STBIR_ARGB_PM,
150 STBIR_ABGR_PM, STBIR_RA_PM or STBIR_AR_PM, you are telling that the pixels
151 ARE premultiplied. In this case, the resizer doesn't have to do the
152 premultipling - it can filter directly on the input. This about twice as
153 fast as the non-premultiplied case, so it's the right option if your data is
154 already setup correctly.
155
156 When you use the pixel layout STBIR_4CHANNEL or STBIR_2CHANNEL, you are
157 telling us that there is no channel that represents transparency; it may be
158 RGB and some unrelated fourth channel that has been stored in the alpha
159 channel, but it is actually not alpha. No special processing will be
160 performed.
161
162 The difference between the generic 4 or 2 channel layouts, and the
163 specialized _PM versions is with the _PM versions you are telling us that
164 the data *is* alpha, just don't premultiply it. That's important when
165 using SRGB pixel formats, we need to know where the alpha is, because
166 it is converted linearly (rather than with the SRGB converters).
167
168 Because alpha weighting produces the same effect as premultiplying, you
169 even have the option with non-premultiplied inputs to let the resizer
170 produce a premultiplied output. Because the intially computed alpha-weighted
171 output image is effectively premultiplied, this is actually more performant
172 than the normal path which un-premultiplies the output image as a final step.
173
174 Finally, when converting both in and out of non-premulitplied space (for
175 example, when using STBIR_RGBA), we go to somewhat heroic measures to
176 ensure that areas with zero alpha value pixels get something reasonable
177 in the RGB values. If you don't care about the RGB values of zero alpha
178 pixels, you can call the stbir_set_non_pm_alpha_speed_over_quality()
179 function - this runs a premultiplied resize about 25% faster. That said,
180 when you really care about speed, using premultiplied pixels for both in
181 and out (STBIR_RGBA_PM, etc) much faster than both of these premultiplied
182 options.
183
184 PIXEL LAYOUT CONVERSION
185 The resizer can convert from some pixel layouts to others. When using the
186 stbir_set_pixel_layouts(), you can, for example, specify STBIR_RGBA
187 on input, and STBIR_ARGB on output, and it will re-organize the channels
188 during the resize. Currently, you can only convert between two pixel
189 layouts with the same number of channels.
190
191 DETERMINISM
192 We commit to being deterministic (from x64 to ARM to scalar to SIMD, etc).
193 This requires compiling with fast-math off (using at least /fp:precise).
194 Also, you must turn off fp-contracting (which turns mult+adds into fmas)!
195 We attempt to do this with pragmas, but with Clang, you usually want to add
196 -ffp-contract=off to the command line as well.
197
198 For 32-bit x86, you must use SSE and SSE2 codegen for determinism. That is,
199 if the scalar x87 unit gets used at all, we immediately lose determinism.
200 On Microsoft Visual Studio 2008 and earlier, from what we can tell there is
201 no way to be deterministic in 32-bit x86 (some x87 always leaks in, even
202 with fp:strict). On 32-bit x86 GCC, determinism requires both -msse2 and
203 -fpmath=sse.
204
205 Note that we will not be deterministic with float data containing NaNs -
206 the NaNs will propagate differently on different SIMD and platforms.
207
208 If you turn on STBIR_USE_FMA, then we will be deterministic with other
209 fma targets, but we will differ from non-fma targets (this is unavoidable,
210 because a fma isn't simply an add with a mult - it also introduces a
211 rounding difference compared to non-fma instruction sequences.
212
213 FLOAT PIXEL FORMAT RANGE
214 Any range of values can be used for the non-alpha float data that you pass
215 in (0 to 1, -1 to 1, whatever). However, if you are inputting float values
216 but *outputting* bytes or shorts, you must use a range of 0 to 1 so that we
217 scale back properly. The alpha channel must also be 0 to 1 for any format
218 that does premultiplication prior to resizing.
219
220 Note also that with float output, using filters with negative lobes, the
221 output filtered values might go slightly out of range. You can define
222 STBIR_FLOAT_LOW_CLAMP and/or STBIR_FLOAT_HIGH_CLAMP to specify the range
223 to clamp to on output, if that's important.
224
225 MAX/MIN SCALE FACTORS
226 The input pixel resolutions are in integers, and we do the internal pointer
227 resolution in size_t sized integers. However, the scale ratio from input
228 resolution to output resolution is calculated in float form. This means
229 the effective possible scale ratio is limited to 24 bits (or 16 million
230 to 1). As you get close to the size of the float resolution (again, 16
231 million pixels wide or high), you might start seeing float inaccuracy
232 issues in general in the pipeline. If you have to do extreme resizes,
233 you can usually do this is multiple stages (using float intermediate
234 buffers).
235
236 FLIPPED IMAGES
237 Stride is just the delta from one scanline to the next. This means you can
238 use a negative stride to handle inverted images (point to the final
239 scanline and use a negative stride). You can invert the input or output,
240 using negative strides.
241
242 DEFAULT FILTERS
243 For functions which don't provide explicit control over what filters to
244 use, you can change the compile-time defaults with:
245
246 #define STBIR_DEFAULT_FILTER_UPSAMPLE STBIR_FILTER_something
247 #define STBIR_DEFAULT_FILTER_DOWNSAMPLE STBIR_FILTER_something
248
249 See stbir_filter in the header-file section for the list of filters.
250
251 NEW FILTERS
252 A number of 1D filter kernels are supplied. For a list of supported
253 filters, see the stbir_filter enum. You can install your own filters by
254 using the stbir_set_filter_callbacks function.
255
256 PROGRESS
257 For interactive use with slow resize operations, you can use the
258 scanline callbacks in the extended API. It would have to be a *very* large
259 image resample to need progress though - we're very fast.
260
261 CEIL and FLOOR
262 In scalar mode, the only functions we use from math.h are ceilf and floorf,
263 but if you have your own versions, you can define the STBIR_CEILF(v) and
264 STBIR_FLOORF(v) macros and we'll use them instead. In SIMD, we just use
265 our own versions.
266
267 ASSERT
268 Define STBIR_ASSERT(boolval) to override assert() and not use assert.h
269
270 PORTING FROM VERSION 1
271 The API has changed. You can continue to use the old version of stb_image_resize.h,
272 which is available in the "deprecated/" directory.
273
274 If you're using the old simple-to-use API, porting is straightforward.
275 (For more advanced APIs, read the documentation.)
276
277 stbir_resize_uint8():
278 - call `stbir_resize_uint8_linear`, cast channel count to `stbir_pixel_layout`
279
280 stbir_resize_float():
281 - call `stbir_resize_float_linear`, cast channel count to `stbir_pixel_layout`
282
283 stbir_resize_uint8_srgb():
284 - function name is unchanged
285 - cast channel count to `stbir_pixel_layout`
286 - above is sufficient unless your image has alpha and it's not RGBA/BGRA
287 - in that case, follow the below instructions for stbir_resize_uint8_srgb_edgemode
288
289 stbir_resize_uint8_srgb_edgemode()
290 - switch to the "medium complexity" API
291 - stbir_resize(), very similar API but a few more parameters:
292 - pixel_layout: cast channel count to `stbir_pixel_layout`
293 - data_type: STBIR_TYPE_UINT8_SRGB
294 - edge: unchanged (STBIR_EDGE_WRAP, etc.)
295 - filter: STBIR_FILTER_DEFAULT
296 - which channel is alpha is specified in stbir_pixel_layout, see enum for details
297
298 FUTURE TODOS
299 * For polyphase integral filters, we just memcpy the coeffs to dupe
300 them, but we should indirect and use the same coeff memory.
301 * Add pixel layout conversions for sensible different channel counts
302 (maybe, 1->3/4, 3->4, 4->1, 3->1).
303 * For SIMD encode and decode scanline routines, do any pre-aligning
304 for bad input/output buffer alignments and pitch?
305 * For very wide scanlines, we should we do vertical strips to stay within
306 L2 cache. Maybe do chunks of 1K pixels at a time. There would be
307 some pixel reconversion, but probably dwarfed by things falling out
308 of cache. Probably also something possible with alternating between
309 scattering and gathering at high resize scales?
310 * Should we have a multiple MIPs at the same time function (could keep
311 more memory in cache during multiple resizes)?
312 * Rewrite the coefficient generator to do many at once.
313 * AVX-512 vertical kernels - worried about downclocking here.
314 * Convert the reincludes to macros when we know they aren't changing.
315 * Experiment with pivoting the horizontal and always using the
316 vertical filters (which are faster, but perhaps not enough to overcome
317 the pivot cost and the extra memory touches). Need to buffer the whole
318 image so have to balance memory use.
319 * Most of our code is internally function pointers, should we compile
320 all the SIMD stuff always and dynamically dispatch?
321
322 CONTRIBUTORS
323 Jeff Roberts: 2.0 implementation, optimizations, SIMD
324 Martins Mozeiko: NEON simd, WASM simd, clang and GCC whisperer
325 Fabian Giesen: half float and srgb converters
326 Sean Barrett: API design, optimizations
327 Jorge L Rodriguez: Original 1.0 implementation
328 Aras Pranckevicius: bugfixes
329 Nathan Reed: warning fixes for 1.0
330
331 REVISIONS
332 2.18 (2026-03-25) fixed coefficient calculation when skipping a coefficient off
333 the left side of the window, added non-aligned access safe
334 memcpy mode for scalar path, fixed various typos, and fixed
335 define error in the float clamp output mode.
336 2.17 (2025-10-25) silly format bug in easy-to-use APIs.
337 2.16 (2025-10-21) fixed the easy-to-use APIs to allow inverted bitmaps (negative
338 strides), fix vertical filter kernel callback, fix threaded
339 gather buffer priming (and assert).
340 (thanks adipose, TainZerL, and Harrison Green)
341 2.15 (2025-07-17) fixed an assert in debug mode when using floats with input
342 callbacks, work around GCC warning when adding to null ptr
343 (thanks Johannes Spohr and Pyry Kovanen).
344 2.14 (2025-05-09) fixed a bug using downsampling gather horizontal first, and
345 scatter with vertical first.
346 2.13 (2025-02-27) fixed a bug when using input callbacks, turned off simd for
347 tiny-c, fixed some variables that should have been static,
348 fixes a bug when calculating temp memory with resizes that
349 exceed 2GB of temp memory (very large resizes).
350 2.12 (2024-10-18) fix incorrect use of user_data with STBIR_FREE
351 2.11 (2024-09-08) fix harmless asan warnings in 2-channel and 3-channel mode
352 with AVX-2, fix some weird scaling edge conditions with
353 point sample mode.
354 2.10 (2024-07-27) fix the defines GCC and mingw for loop unroll control,
355 fix MSVC 32-bit arm half float routines.
356 2.09 (2024-06-19) fix the defines for 32-bit ARM GCC builds (was selecting
357 hardware half floats).
358 2.08 (2024-06-10) fix for RGB->BGR three channel flips and add SIMD (thanks
359 to Ryan Salsbury), fix for sub-rect resizes, use the
360 pragmas to control unrolling when they are available.
361 2.07 (2024-05-24) fix for slow final split during threaded conversions of very
362 wide scanlines when downsampling (caused by extra input
363 converting), fix for wide scanline resamples with many
364 splits (int overflow), fix GCC warning.
365 2.06 (2024-02-10) fix for identical width/height 3x or more down-scaling
366 undersampling a single row on rare resize ratios (about 1%).
367 2.05 (2024-02-07) fix for 2 pixel to 1 pixel resizes with wrap (thanks Aras),
368 fix for output callback (thanks Julien Koenen).
369 2.04 (2023-11-17) fix for rare AVX bug, shadowed symbol (thanks Nikola Smiljanic).
370 2.03 (2023-11-01) ASAN and TSAN warnings fixed, minor tweaks.
371 2.00 (2023-10-10) mostly new source: new api, optimizations, simd, vertical-first, etc
372 2x-5x faster without simd, 4x-12x faster with simd,
373 in some cases, 20x to 40x faster esp resizing large to very small.
374 0.96 (2019-03-04) fixed warnings
375 0.95 (2017-07-23) fixed warnings
376 0.94 (2017-03-18) fixed warnings
377 0.93 (2017-03-03) fixed bug with certain combinations of heights
378 0.92 (2017-01-02) fix integer overflow on large (>2GB) images
379 0.91 (2016-04-02) fix warnings; fix handling of subpixel regions
380 0.90 (2014-09-17) first released version
381
382 LICENSE
383 See end of file for license information.
384*/
385
386#if !defined(STB_IMAGE_RESIZE_DO_HORIZONTALS) && !defined(STB_IMAGE_RESIZE_DO_VERTICALS) && !defined(STB_IMAGE_RESIZE_DO_CODERS) // for internal re-includes
387
388#ifndef STBIR_INCLUDE_STB_IMAGE_RESIZE2_H
389#define STBIR_INCLUDE_STB_IMAGE_RESIZE2_H
390
391#include <stddef.h>
392#ifdef _MSC_VER
393typedef unsigned char stbir_uint8;
394typedef unsigned short stbir_uint16;
395typedef unsigned int stbir_uint32;
396typedef unsigned __int64 stbir_uint64;
397#else
398#include <stdint.h>
399typedef uint8_t stbir_uint8;
400typedef uint16_t stbir_uint16;
401typedef uint32_t stbir_uint32;
402typedef uint64_t stbir_uint64;
403#endif
404
405#ifndef STBIRDEF
406#ifdef STB_IMAGE_RESIZE_STATIC
407#define STBIRDEF static
408#else
409#ifdef __cplusplus
410#define STBIRDEF extern "C"
411#else
412#define STBIRDEF extern
413#endif
414#endif
415#endif
416
417//////////////////////////////////////////////////////////////////////////////
418//// start "header file" ///////////////////////////////////////////////////
419//
420// Easy-to-use API:
421//
422// * stride is the offset between successive rows of image data
423// in memory, in bytes. specify 0 for packed continuously in memory
424// * colorspace is linear or sRGB as specified by function name
425// * Uses the default filters
426// * Uses edge mode clamped
427// * returned result is 1 for success or 0 in case of an error.
428
429
430// stbir_pixel_layout specifies:
431// number of channels
432// order of channels
433// whether color is premultiplied by alpha
434// for back compatibility, you can cast the old channel count to an stbir_pixel_layout
435typedef enum
436{
437 STBIR_1CHANNEL = 1,
438 STBIR_2CHANNEL = 2,
439 STBIR_RGB = 3, // 3-chan, with order specified (for channel flipping)
440 STBIR_BGR = 0, // 3-chan, with order specified (for channel flipping)
441 STBIR_4CHANNEL = 5,
442
443 STBIR_RGBA = 4, // alpha formats, where alpha is NOT premultiplied into color channels
444 STBIR_BGRA = 6,
445 STBIR_ARGB = 7,
446 STBIR_ABGR = 8,
447 STBIR_RA = 9,
448 STBIR_AR = 10,
449
450 STBIR_RGBA_PM = 11, // alpha formats, where alpha is premultiplied into color channels
451 STBIR_BGRA_PM = 12,
452 STBIR_ARGB_PM = 13,
453 STBIR_ABGR_PM = 14,
454 STBIR_RA_PM = 15,
455 STBIR_AR_PM = 16,
456
457 STBIR_RGBA_NO_AW = 11, // alpha formats, where NO alpha weighting is applied at all!
458 STBIR_BGRA_NO_AW = 12, // these are just synonyms for the _PM flags (which also do
459 STBIR_ARGB_NO_AW = 13, // no alpha weighting). These names just make it more clear
460 STBIR_ABGR_NO_AW = 14, // for some folks).
461 STBIR_RA_NO_AW = 15,
462 STBIR_AR_NO_AW = 16,
463
464} stbir_pixel_layout;
465
466//===============================================================
467// Simple-complexity API
468//
469// If output_pixels is NULL (0), then we will allocate the buffer and return it to you.
470//--------------------------------
471
472STBIRDEF unsigned char * stbir_resize_uint8_srgb( const unsigned char *input_pixels , int input_w , int input_h, int input_stride_in_bytes,
473 unsigned char *output_pixels, int output_w, int output_h, int output_stride_in_bytes,
474 stbir_pixel_layout pixel_type );
475
476STBIRDEF unsigned char * stbir_resize_uint8_linear( const unsigned char *input_pixels , int input_w , int input_h, int input_stride_in_bytes,
477 unsigned char *output_pixels, int output_w, int output_h, int output_stride_in_bytes,
478 stbir_pixel_layout pixel_type );
479
480STBIRDEF float * stbir_resize_float_linear( const float *input_pixels , int input_w , int input_h, int input_stride_in_bytes,
481 float *output_pixels, int output_w, int output_h, int output_stride_in_bytes,
482 stbir_pixel_layout pixel_type );
483//===============================================================
484
485//===============================================================
486// Medium-complexity API
487//
488// This extends the easy-to-use API as follows:
489//
490// * Can specify the datatype - U8, U8_SRGB, U16, FLOAT, HALF_FLOAT
491// * Edge wrap can selected explicitly
492// * Filter can be selected explicitly
493//--------------------------------
494
495typedef enum
496{
497 STBIR_EDGE_CLAMP = 0,
498 STBIR_EDGE_REFLECT = 1,
499 STBIR_EDGE_WRAP = 2, // this edge mode is slower and uses more memory
500 STBIR_EDGE_ZERO = 3,
501} stbir_edge;
502
503typedef enum
504{
505 STBIR_FILTER_DEFAULT = 0, // use same filter type that easy-to-use API chooses
506 STBIR_FILTER_BOX = 1, // A trapezoid w/1-pixel wide ramps, same result as box for integer scale ratios
507 STBIR_FILTER_TRIANGLE = 2, // On upsampling, produces same results as bilinear texture filtering
508 STBIR_FILTER_CUBICBSPLINE = 3, // The cubic b-spline (aka Mitchell-Netrevalli with B=1,C=0), gaussian-esque
509 STBIR_FILTER_CATMULLROM = 4, // An interpolating cubic spline
510 STBIR_FILTER_MITCHELL = 5, // Mitchell-Netrevalli filter with B=1/3, C=1/3
511 STBIR_FILTER_POINT_SAMPLE = 6, // Simple point sampling
512 STBIR_FILTER_OTHER = 7, // User callback specified
513} stbir_filter;
514
515typedef enum
516{
517 STBIR_TYPE_UINT8 = 0,
518 STBIR_TYPE_UINT8_SRGB = 1,
519 STBIR_TYPE_UINT8_SRGB_ALPHA = 2, // alpha channel, when present, should also be SRGB (this is very unusual)
520 STBIR_TYPE_UINT16 = 3,
521 STBIR_TYPE_FLOAT = 4,
522 STBIR_TYPE_HALF_FLOAT = 5
523} stbir_datatype;
524
525// medium api
526STBIRDEF void * stbir_resize( const void *input_pixels , int input_w , int input_h, int input_stride_in_bytes,
527 void *output_pixels, int output_w, int output_h, int output_stride_in_bytes,
528 stbir_pixel_layout pixel_layout, stbir_datatype data_type,
529 stbir_edge edge, stbir_filter filter );
530//===============================================================
531
532
533
534//===============================================================
535// Extended-complexity API
536//
537// This API exposes all resize functionality.
538//
539// * Separate filter types for each axis
540// * Separate edge modes for each axis
541// * Separate input and output data types
542// * Can specify regions with subpixel correctness
543// * Can specify alpha flags
544// * Can specify a memory callback
545// * Can specify a callback data type for pixel input and output
546// * Can be threaded for a single resize
547// * Can be used to resize many frames without recalculating the sampler info
548//
549// Use this API as follows:
550// 1) Call the stbir_resize_init function on a local STBIR_RESIZE structure
551// 2) Call any of the stbir_set functions
552// 3) Optionally call stbir_build_samplers() if you are going to resample multiple times
553// with the same input and output dimensions (like resizing video frames)
554// 4) Resample by calling stbir_resize_extended().
555// 5) Call stbir_free_samplers() if you called stbir_build_samplers()
556//--------------------------------
557
558
559// Types:
560
561// INPUT CALLBACK: this callback is used for input scanlines
562typedef void const * stbir_input_callback( void * optional_output, void const * input_ptr, int num_pixels, int x, int y, void * context );
563
564// OUTPUT CALLBACK: this callback is used for output scanlines
565typedef void stbir_output_callback( void const * output_ptr, int num_pixels, int y, void * context );
566
567// callbacks for user installed filters
568typedef float stbir__kernel_callback( float x, float scale, void * user_data ); // centered at zero
569typedef float stbir__support_callback( float scale, void * user_data );
570
571// internal structure with precomputed scaling
572typedef struct stbir__info stbir__info;
573
574typedef struct STBIR_RESIZE // use the stbir_resize_init and stbir_override functions to set these values for future compatibility
575{
576 void * user_data;
577 void const * input_pixels;
578 int input_w, input_h;
579 double input_s0, input_t0, input_s1, input_t1;
580 stbir_input_callback * input_cb;
581 void * output_pixels;
582 int output_w, output_h;
583 int output_subx, output_suby, output_subw, output_subh;
584 stbir_output_callback * output_cb;
585 int input_stride_in_bytes;
586 int output_stride_in_bytes;
587 int splits;
588 int fast_alpha;
589 int needs_rebuild;
590 int called_alloc;
591 stbir_pixel_layout input_pixel_layout_public;
592 stbir_pixel_layout output_pixel_layout_public;
593 stbir_datatype input_data_type;
594 stbir_datatype output_data_type;
595 stbir_filter horizontal_filter, vertical_filter;
596 stbir_edge horizontal_edge, vertical_edge;
597 stbir__kernel_callback * horizontal_filter_kernel; stbir__support_callback * horizontal_filter_support;
598 stbir__kernel_callback * vertical_filter_kernel; stbir__support_callback * vertical_filter_support;
599 stbir__info * samplers;
600} STBIR_RESIZE;
601
602// extended complexity api
603
604
605// First off, you must ALWAYS call stbir_resize_init on your resize structure before any of the other calls!
606STBIRDEF void stbir_resize_init( STBIR_RESIZE * resize,
607 const void *input_pixels, int input_w, int input_h, int input_stride_in_bytes, // stride can be zero
608 void *output_pixels, int output_w, int output_h, int output_stride_in_bytes, // stride can be zero
609 stbir_pixel_layout pixel_layout, stbir_datatype data_type );
610
611//===============================================================
612// You can update these parameters any time after resize_init and there is no cost
613//--------------------------------
614
615STBIRDEF void stbir_set_datatypes( STBIR_RESIZE * resize, stbir_datatype input_type, stbir_datatype output_type );
616STBIRDEF void stbir_set_pixel_callbacks( STBIR_RESIZE * resize, stbir_input_callback * input_cb, stbir_output_callback * output_cb ); // no callbacks by default
617STBIRDEF void stbir_set_user_data( STBIR_RESIZE * resize, void * user_data ); // pass back STBIR_RESIZE* by default
618STBIRDEF void stbir_set_buffer_ptrs( STBIR_RESIZE * resize, const void * input_pixels, int input_stride_in_bytes, void * output_pixels, int output_stride_in_bytes );
619
620//===============================================================
621
622
623//===============================================================
624// If you call any of these functions, you will trigger a sampler rebuild!
625//--------------------------------
626
627STBIRDEF int stbir_set_pixel_layouts( STBIR_RESIZE * resize, stbir_pixel_layout input_pixel_layout, stbir_pixel_layout output_pixel_layout ); // sets new buffer layouts
628STBIRDEF int stbir_set_edgemodes( STBIR_RESIZE * resize, stbir_edge horizontal_edge, stbir_edge vertical_edge ); // CLAMP by default
629
630STBIRDEF int stbir_set_filters( STBIR_RESIZE * resize, stbir_filter horizontal_filter, stbir_filter vertical_filter ); // STBIR_DEFAULT_FILTER_UPSAMPLE/DOWNSAMPLE by default
631STBIRDEF int stbir_set_filter_callbacks( STBIR_RESIZE * resize, stbir__kernel_callback * horizontal_filter, stbir__support_callback * horizontal_support, stbir__kernel_callback * vertical_filter, stbir__support_callback * vertical_support );
632
633STBIRDEF int stbir_set_pixel_subrect( STBIR_RESIZE * resize, int subx, int suby, int subw, int subh ); // sets both sub-regions (full regions by default)
634STBIRDEF int stbir_set_input_subrect( STBIR_RESIZE * resize, double s0, double t0, double s1, double t1 ); // sets input sub-region (full region by default)
635STBIRDEF int stbir_set_output_pixel_subrect( STBIR_RESIZE * resize, int subx, int suby, int subw, int subh ); // sets output sub-region (full region by default)
636
637// when inputting AND outputting non-premultiplied alpha pixels, we use a slower but higher quality technique
638// that fills the zero alpha pixel's RGB values with something plausible. If you don't care about areas of
639// zero alpha, you can call this function to get about a 25% speed improvement for STBIR_RGBA to STBIR_RGBA
640// types of resizes.
641STBIRDEF int stbir_set_non_pm_alpha_speed_over_quality( STBIR_RESIZE * resize, int non_pma_alpha_speed_over_quality );
642//===============================================================
643
644
645//===============================================================
646// You can call build_samplers to prebuild all the internal data we need to resample.
647// Then, if you call resize_extended many times with the same resize, you only pay the
648// cost once.
649// If you do call build_samplers, you MUST call free_samplers eventually.
650//--------------------------------
651
652// This builds the samplers and does one allocation
653STBIRDEF int stbir_build_samplers( STBIR_RESIZE * resize );
654
655// You MUST call this, if you call stbir_build_samplers or stbir_build_samplers_with_splits
656STBIRDEF void stbir_free_samplers( STBIR_RESIZE * resize );
657//===============================================================
658
659
660// And this is the main function to perform the resize synchronously on one thread.
661STBIRDEF int stbir_resize_extended( STBIR_RESIZE * resize );
662
663
664//===============================================================
665// Use these functions for multithreading.
666// 1) You call stbir_build_samplers_with_splits first on the main thread
667// 2) Then stbir_resize_with_split on each thread
668// 3) stbir_free_samplers when done on the main thread
669//--------------------------------
670
671// This will build samplers for threading.
672// You can pass in the number of threads you'd like to use (try_splits).
673// It returns the number of splits (threads) that you can call it with.
674/// It might be less if the image resize can't be split up that many ways.
675
676STBIRDEF int stbir_build_samplers_with_splits( STBIR_RESIZE * resize, int try_splits );
677
678// This function does a split of the resizing (you call this fuction for each
679// split, on multiple threads). A split is a piece of the output resize pixel space.
680
681// Note that you MUST call stbir_build_samplers_with_splits before stbir_resize_extended_split!
682
683// Usually, you will always call stbir_resize_split with split_start as the thread_index
684// and "1" for the split_count.
685// But, if you have a weird situation where you MIGHT want 8 threads, but sometimes
686// only 4 threads, you can use 0,2,4,6 for the split_start's and use "2" for the
687// split_count each time to turn in into a 4 thread resize. (This is unusual).
688
689STBIRDEF int stbir_resize_extended_split( STBIR_RESIZE * resize, int split_start, int split_count );
690//===============================================================
691
692
693//===============================================================
694// Pixel Callbacks info:
695//--------------------------------
696
697// The input callback is super flexible - it calls you with the input address
698// (based on the stride and base pointer), it gives you an optional_output
699// pointer that you can fill, or you can just return your own pointer into
700// your own data.
701//
702// You can also do conversion from non-supported data types if necessary - in
703// this case, you ignore the input_ptr and just use the x and y parameters to
704// calculate your own input_ptr based on the size of each non-supported pixel.
705// (Something like the third example below.)
706//
707// You can also install just an input or just an output callback by setting the
708// callback that you don't want to zero.
709//
710// First example, progress: (getting a callback that you can monitor the progress):
711// void const * my_callback( void * optional_output, void const * input_ptr, int num_pixels, int x, int y, void * context )
712// {
713// percentage_done = y / input_height;
714// return input_ptr; // use buffer from call
715// }
716//
717// Next example, copying: (copy from some other buffer or stream):
718// void const * my_callback( void * optional_output, void const * input_ptr, int num_pixels, int x, int y, void * context )
719// {
720// CopyOrStreamData( optional_output, other_data_src, num_pixels * pixel_width_in_bytes );
721// return optional_output; // return the optional buffer that we filled
722// }
723//
724// Third example, input another buffer without copying: (zero-copy from other buffer):
725// void const * my_callback( void * optional_output, void const * input_ptr, int num_pixels, int x, int y, void * context )
726// {
727// void * pixels = ( (char*) other_image_base ) + ( y * other_image_stride ) + ( x * other_pixel_width_in_bytes );
728// return pixels; // return pointer to your data without copying
729// }
730//
731//
732// The output callback is considerably simpler - it just calls you so that you can dump
733// out each scanline. You could even directly copy out to disk if you have a simple format
734// like TGA or BMP. You can also convert to other output types here if you want.
735//
736// Simple example:
737// void const * my_output( void * output_ptr, int num_pixels, int y, void * context )
738// {
739// percentage_done = y / output_height;
740// fwrite( output_ptr, pixel_width_in_bytes, num_pixels, output_file );
741// }
742//===============================================================
743
744
745
746
747//===============================================================
748// optional built-in profiling API
749//--------------------------------
750
751#ifdef STBIR_PROFILE
752
753typedef struct STBIR_PROFILE_INFO
754{
755 stbir_uint64 total_clocks;
756
757 // how many clocks spent (of total_clocks) in the various resize routines, along with a string description
758 // there are "resize_count" number of zones
759 stbir_uint64 clocks[ 8 ];
760 char const ** descriptions;
761
762 // count of clocks and descriptions
763 stbir_uint32 count;
764} STBIR_PROFILE_INFO;
765
766// use after calling stbir_resize_extended (or stbir_build_samplers or stbir_build_samplers_with_splits)
767STBIRDEF void stbir_resize_build_profile_info( STBIR_PROFILE_INFO * out_info, STBIR_RESIZE const * resize );
768
769// use after calling stbir_resize_extended
770STBIRDEF void stbir_resize_extended_profile_info( STBIR_PROFILE_INFO * out_info, STBIR_RESIZE const * resize );
771
772// use after calling stbir_resize_extended_split
773STBIRDEF void stbir_resize_split_profile_info( STBIR_PROFILE_INFO * out_info, STBIR_RESIZE const * resize, int split_start, int split_num );
774
775//===============================================================
776
777#endif
778
779
780//// end header file /////////////////////////////////////////////////////
781#endif // STBIR_INCLUDE_STB_IMAGE_RESIZE2_H
782
783#if defined(STB_IMAGE_RESIZE_IMPLEMENTATION) || defined(STB_IMAGE_RESIZE2_IMPLEMENTATION)
784
785#ifndef STBIR_ASSERT
786#include <assert.h>
787#define STBIR_ASSERT(x) assert(x)
788#endif
789
790#ifndef STBIR_MALLOC
791#include <stdlib.h>
792#define STBIR_MALLOC(size,user_data) ((void)(user_data), malloc(size))
793#define STBIR_FREE(ptr,user_data) ((void)(user_data), free(ptr))
794// (we used the comma operator to evaluate user_data, to avoid "unused parameter" warnings)
795#endif
796
797#ifdef _MSC_VER
798
799#define stbir__inline __forceinline
800
801#else
802
803#define stbir__inline __inline__
804
805// Clang address sanitizer
806#if defined(__has_feature)
807 #if __has_feature(address_sanitizer) || __has_feature(memory_sanitizer)
808 #ifndef STBIR__SEPARATE_ALLOCATIONS
809 #define STBIR__SEPARATE_ALLOCATIONS
810 #endif
811 #endif
812#endif
813
814#endif
815
816// GCC and MSVC
817#if defined(__SANITIZE_ADDRESS__)
818 #ifndef STBIR__SEPARATE_ALLOCATIONS
819 #define STBIR__SEPARATE_ALLOCATIONS
820 #endif
821#endif
822
823// Always turn off automatic FMA use - use STBIR_USE_FMA if you want.
824// Otherwise, this is a determinism disaster.
825#ifndef STBIR_DONT_CHANGE_FP_CONTRACT // override in case you don't want this behavior
826#if defined(_MSC_VER) && !defined(__clang__)
827#if _MSC_VER > 1200
828#pragma fp_contract(off)
829#endif
830#elif defined(__GNUC__) && !defined(__clang__)
831#pragma GCC optimize("fp-contract=off")
832#else
833#pragma STDC FP_CONTRACT OFF
834#endif
835#endif
836
837#ifdef _MSC_VER
838#define STBIR__UNUSED(v) (void)(v)
839#else
840#define STBIR__UNUSED(v) (void)sizeof(v)
841#endif
842
843#define STBIR__ARRAY_SIZE(a) (sizeof((a))/sizeof((a)[0]))
844
845
846#ifndef STBIR_DEFAULT_FILTER_UPSAMPLE
847#define STBIR_DEFAULT_FILTER_UPSAMPLE STBIR_FILTER_CATMULLROM
848#endif
849
850#ifndef STBIR_DEFAULT_FILTER_DOWNSAMPLE
851#define STBIR_DEFAULT_FILTER_DOWNSAMPLE STBIR_FILTER_MITCHELL
852#endif
853
854
855#ifndef STBIR__HEADER_FILENAME
856#define STBIR__HEADER_FILENAME "stb_image_resize2.h"
857#endif
858
859// the internal pixel layout enums are in a different order, so we can easily do range comparisons of types
860// the public pixel layout is ordered in a way that if you cast num_channels (1-4) to the enum, you get something sensible
861typedef enum
862{
863 STBIRI_1CHANNEL = 0,
864 STBIRI_2CHANNEL = 1,
865 STBIRI_RGB = 2,
866 STBIRI_BGR = 3,
867 STBIRI_4CHANNEL = 4,
868
869 STBIRI_RGBA = 5,
870 STBIRI_BGRA = 6,
871 STBIRI_ARGB = 7,
872 STBIRI_ABGR = 8,
873 STBIRI_RA = 9,
874 STBIRI_AR = 10,
875
876 STBIRI_RGBA_PM = 11,
877 STBIRI_BGRA_PM = 12,
878 STBIRI_ARGB_PM = 13,
879 STBIRI_ABGR_PM = 14,
880 STBIRI_RA_PM = 15,
881 STBIRI_AR_PM = 16,
882} stbir_internal_pixel_layout;
883
884// define the public pixel layouts to not compile inside the implementation (to avoid accidental use)
885#define STBIR_BGR bad_dont_use_in_implementation
886#define STBIR_1CHANNEL STBIR_BGR
887#define STBIR_2CHANNEL STBIR_BGR
888#define STBIR_RGB STBIR_BGR
889#define STBIR_RGBA STBIR_BGR
890#define STBIR_4CHANNEL STBIR_BGR
891#define STBIR_BGRA STBIR_BGR
892#define STBIR_ARGB STBIR_BGR
893#define STBIR_ABGR STBIR_BGR
894#define STBIR_RA STBIR_BGR
895#define STBIR_AR STBIR_BGR
896#define STBIR_RGBA_PM STBIR_BGR
897#define STBIR_BGRA_PM STBIR_BGR
898#define STBIR_ARGB_PM STBIR_BGR
899#define STBIR_ABGR_PM STBIR_BGR
900#define STBIR_RA_PM STBIR_BGR
901#define STBIR_AR_PM STBIR_BGR
902
903// must match stbir_datatype
904static unsigned char stbir__type_size[] = {
905 1,1,1,2,4,2 // STBIR_TYPE_UINT8,STBIR_TYPE_UINT8_SRGB,STBIR_TYPE_UINT8_SRGB_ALPHA,STBIR_TYPE_UINT16,STBIR_TYPE_FLOAT,STBIR_TYPE_HALF_FLOAT
906};
907
908// When gathering, the contributors are which source pixels contribute.
909// When scattering, the contributors are which destination pixels are contributed to.
910typedef struct
911{
912 int n0; // First contributing pixel
913 int n1; // Last contributing pixel
914} stbir__contributors;
915
916typedef struct
917{
918 int lowest; // First sample index for whole filter
919 int highest; // Last sample index for whole filter
920 int widest; // widest single set of samples for an output
921} stbir__filter_extent_info;
922
923typedef struct
924{
925 int n0; // First pixel of decode buffer to write to
926 int n1; // Last pixel of decode that will be written to
927 int pixel_offset_for_input; // Pixel offset into input_scanline
928} stbir__span;
929
930typedef struct stbir__scale_info
931{
932 int input_full_size;
933 int output_sub_size;
934 float scale;
935 float inv_scale;
936 float pixel_shift; // starting shift in output pixel space (in pixels)
937 int scale_is_rational;
938 stbir_uint32 scale_numerator, scale_denominator;
939} stbir__scale_info;
940
941typedef struct
942{
943 stbir__contributors * contributors;
944 float* coefficients;
945 stbir__contributors * gather_prescatter_contributors;
946 float * gather_prescatter_coefficients;
947 stbir__scale_info scale_info;
948 float support;
949 stbir_filter filter_enum;
950 stbir__kernel_callback * filter_kernel;
951 stbir__support_callback * filter_support;
952 stbir_edge edge;
953 int coefficient_width;
954 int filter_pixel_width;
955 int filter_pixel_margin;
956 int num_contributors;
957 int contributors_size;
958 int coefficients_size;
959 stbir__filter_extent_info extent_info;
960 int is_gather; // 0 = scatter, 1 = gather with scale >= 1, 2 = gather with scale < 1
961 int gather_prescatter_num_contributors;
962 int gather_prescatter_coefficient_width;
963 int gather_prescatter_contributors_size;
964 int gather_prescatter_coefficients_size;
965} stbir__sampler;
966
967typedef struct
968{
969 stbir__contributors conservative;
970 int edge_sizes[2]; // this can be less than filter_pixel_margin, if the filter and scaling falls off
971 stbir__span spans[2]; // can be two spans, if doing input subrect with clamp mode WRAP
972} stbir__extents;
973
974typedef struct
975{
976#ifdef STBIR_PROFILE
977 union
978 {
979 struct { stbir_uint64 total, looping, vertical, horizontal, decode, encode, alpha, unalpha; } named;
980 stbir_uint64 array[8];
981 } profile;
982 stbir_uint64 * current_zone_excluded_ptr;
983#endif
984 float* decode_buffer;
985
986 int ring_buffer_first_scanline;
987 int ring_buffer_last_scanline;
988 int ring_buffer_begin_index; // first_scanline is at this index in the ring buffer
989 int start_output_y, end_output_y;
990 int start_input_y, end_input_y; // used in scatter only
991
992 #ifdef STBIR__SEPARATE_ALLOCATIONS
993 float** ring_buffers; // one pointer for each ring buffer
994 #else
995 float* ring_buffer; // one big buffer that we index into
996 #endif
997
998 float* vertical_buffer;
999
1000 char no_cache_straddle[64];
1001} stbir__per_split_info;
1002
1003typedef float * stbir__decode_pixels_func( float * decode, int width_times_channels, void const * input );
1004typedef void stbir__alpha_weight_func( float * decode_buffer, int width_times_channels );
1005typedef void stbir__horizontal_gather_channels_func( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer,
1006 stbir__contributors const * horizontal_contributors, float const * horizontal_coefficients, int coefficient_width );
1007typedef void stbir__alpha_unweight_func(float * encode_buffer, int width_times_channels );
1008typedef void stbir__encode_pixels_func( void * output, int width_times_channels, float const * encode );
1009
1010struct stbir__info
1011{
1012#ifdef STBIR_PROFILE
1013 union
1014 {
1015 struct { stbir_uint64 total, build, alloc, horizontal, vertical, cleanup, pivot; } named;
1016 stbir_uint64 array[7];
1017 } profile;
1018 stbir_uint64 * current_zone_excluded_ptr;
1019#endif
1020 stbir__sampler horizontal;
1021 stbir__sampler vertical;
1022
1023 void const * input_data;
1024 void * output_data;
1025
1026 int input_stride_bytes;
1027 int output_stride_bytes;
1028 int ring_buffer_length_bytes; // The length of an individual entry in the ring buffer. The total number of ring buffers is stbir__get_filter_pixel_width(filter)
1029 int ring_buffer_num_entries; // Total number of entries in the ring buffer.
1030
1031 stbir_datatype input_type;
1032 stbir_datatype output_type;
1033
1034 stbir_input_callback * in_pixels_cb;
1035 void * user_data;
1036 stbir_output_callback * out_pixels_cb;
1037
1038 stbir__extents scanline_extents;
1039
1040 void * alloced_mem;
1041 stbir__per_split_info * split_info; // by default 1, but there will be N of these allocated based on the thread init you did
1042
1043 stbir__decode_pixels_func * decode_pixels;
1044 stbir__alpha_weight_func * alpha_weight;
1045 stbir__horizontal_gather_channels_func * horizontal_gather_channels;
1046 stbir__alpha_unweight_func * alpha_unweight;
1047 stbir__encode_pixels_func * encode_pixels;
1048
1049 int alloc_ring_buffer_num_entries; // Number of entries in the ring buffer that will be allocated
1050 int splits; // count of splits
1051
1052 stbir_internal_pixel_layout input_pixel_layout_internal;
1053 stbir_internal_pixel_layout output_pixel_layout_internal;
1054
1055 int input_color_and_type;
1056 int offset_x, offset_y; // offset within output_data
1057 int vertical_first;
1058 int channels;
1059 int effective_channels; // same as channels, except on RGBA/ARGB (7), or XA/AX (3)
1060 size_t alloced_total;
1061};
1062
1063
1064#define stbir__max_uint8_as_float 255.0f
1065#define stbir__max_uint16_as_float 65535.0f
1066#define stbir__max_uint8_as_float_inverted 3.9215689e-03f // (1.0f/255.0f)
1067#define stbir__max_uint16_as_float_inverted 1.5259022e-05f // (1.0f/65535.0f)
1068#define stbir__small_float ((float)1 / (1 << 20) / (1 << 20) / (1 << 20) / (1 << 20) / (1 << 20) / (1 << 20))
1069
1070// min/max friendly
1071#define STBIR_CLAMP(x, xmin, xmax) for(;;) { \
1072 if ( (x) < (xmin) ) (x) = (xmin); \
1073 if ( (x) > (xmax) ) (x) = (xmax); \
1074 break; \
1075}
1076
1077static stbir__inline int stbir__min(int a, int b)
1078{
1079 return a < b ? a : b;
1080}
1081
1082static stbir__inline int stbir__max(int a, int b)
1083{
1084 return a > b ? a : b;
1085}
1086
1087static float stbir__srgb_uchar_to_linear_float[256] = {
1088 0.000000f, 0.000304f, 0.000607f, 0.000911f, 0.001214f, 0.001518f, 0.001821f, 0.002125f, 0.002428f, 0.002732f, 0.003035f,
1089 0.003347f, 0.003677f, 0.004025f, 0.004391f, 0.004777f, 0.005182f, 0.005605f, 0.006049f, 0.006512f, 0.006995f, 0.007499f,
1090 0.008023f, 0.008568f, 0.009134f, 0.009721f, 0.010330f, 0.010960f, 0.011612f, 0.012286f, 0.012983f, 0.013702f, 0.014444f,
1091 0.015209f, 0.015996f, 0.016807f, 0.017642f, 0.018500f, 0.019382f, 0.020289f, 0.021219f, 0.022174f, 0.023153f, 0.024158f,
1092 0.025187f, 0.026241f, 0.027321f, 0.028426f, 0.029557f, 0.030713f, 0.031896f, 0.033105f, 0.034340f, 0.035601f, 0.036889f,
1093 0.038204f, 0.039546f, 0.040915f, 0.042311f, 0.043735f, 0.045186f, 0.046665f, 0.048172f, 0.049707f, 0.051269f, 0.052861f,
1094 0.054480f, 0.056128f, 0.057805f, 0.059511f, 0.061246f, 0.063010f, 0.064803f, 0.066626f, 0.068478f, 0.070360f, 0.072272f,
1095 0.074214f, 0.076185f, 0.078187f, 0.080220f, 0.082283f, 0.084376f, 0.086500f, 0.088656f, 0.090842f, 0.093059f, 0.095307f,
1096 0.097587f, 0.099899f, 0.102242f, 0.104616f, 0.107023f, 0.109462f, 0.111932f, 0.114435f, 0.116971f, 0.119538f, 0.122139f,
1097 0.124772f, 0.127438f, 0.130136f, 0.132868f, 0.135633f, 0.138432f, 0.141263f, 0.144128f, 0.147027f, 0.149960f, 0.152926f,
1098 0.155926f, 0.158961f, 0.162029f, 0.165132f, 0.168269f, 0.171441f, 0.174647f, 0.177888f, 0.181164f, 0.184475f, 0.187821f,
1099 0.191202f, 0.194618f, 0.198069f, 0.201556f, 0.205079f, 0.208637f, 0.212231f, 0.215861f, 0.219526f, 0.223228f, 0.226966f,
1100 0.230740f, 0.234551f, 0.238398f, 0.242281f, 0.246201f, 0.250158f, 0.254152f, 0.258183f, 0.262251f, 0.266356f, 0.270498f,
1101 0.274677f, 0.278894f, 0.283149f, 0.287441f, 0.291771f, 0.296138f, 0.300544f, 0.304987f, 0.309469f, 0.313989f, 0.318547f,
1102 0.323143f, 0.327778f, 0.332452f, 0.337164f, 0.341914f, 0.346704f, 0.351533f, 0.356400f, 0.361307f, 0.366253f, 0.371238f,
1103 0.376262f, 0.381326f, 0.386430f, 0.391573f, 0.396755f, 0.401978f, 0.407240f, 0.412543f, 0.417885f, 0.423268f, 0.428691f,
1104 0.434154f, 0.439657f, 0.445201f, 0.450786f, 0.456411f, 0.462077f, 0.467784f, 0.473532f, 0.479320f, 0.485150f, 0.491021f,
1105 0.496933f, 0.502887f, 0.508881f, 0.514918f, 0.520996f, 0.527115f, 0.533276f, 0.539480f, 0.545725f, 0.552011f, 0.558340f,
1106 0.564712f, 0.571125f, 0.577581f, 0.584078f, 0.590619f, 0.597202f, 0.603827f, 0.610496f, 0.617207f, 0.623960f, 0.630757f,
1107 0.637597f, 0.644480f, 0.651406f, 0.658375f, 0.665387f, 0.672443f, 0.679543f, 0.686685f, 0.693872f, 0.701102f, 0.708376f,
1108 0.715694f, 0.723055f, 0.730461f, 0.737911f, 0.745404f, 0.752942f, 0.760525f, 0.768151f, 0.775822f, 0.783538f, 0.791298f,
1109 0.799103f, 0.806952f, 0.814847f, 0.822786f, 0.830770f, 0.838799f, 0.846873f, 0.854993f, 0.863157f, 0.871367f, 0.879622f,
1110 0.887923f, 0.896269f, 0.904661f, 0.913099f, 0.921582f, 0.930111f, 0.938686f, 0.947307f, 0.955974f, 0.964686f, 0.973445f,
1111 0.982251f, 0.991102f, 1.0f
1112};
1113
1114typedef union
1115{
1116 unsigned int u;
1117 float f;
1118} stbir__FP32;
1119
1120// From https://gist.github.com/rygorous/2203834
1121
1122static const stbir_uint32 fp32_to_srgb8_tab4[104] = {
1123 0x0073000d, 0x007a000d, 0x0080000d, 0x0087000d, 0x008d000d, 0x0094000d, 0x009a000d, 0x00a1000d,
1124 0x00a7001a, 0x00b4001a, 0x00c1001a, 0x00ce001a, 0x00da001a, 0x00e7001a, 0x00f4001a, 0x0101001a,
1125 0x010e0033, 0x01280033, 0x01410033, 0x015b0033, 0x01750033, 0x018f0033, 0x01a80033, 0x01c20033,
1126 0x01dc0067, 0x020f0067, 0x02430067, 0x02760067, 0x02aa0067, 0x02dd0067, 0x03110067, 0x03440067,
1127 0x037800ce, 0x03df00ce, 0x044600ce, 0x04ad00ce, 0x051400ce, 0x057b00c5, 0x05dd00bc, 0x063b00b5,
1128 0x06970158, 0x07420142, 0x07e30130, 0x087b0120, 0x090b0112, 0x09940106, 0x0a1700fc, 0x0a9500f2,
1129 0x0b0f01cb, 0x0bf401ae, 0x0ccb0195, 0x0d950180, 0x0e56016e, 0x0f0d015e, 0x0fbc0150, 0x10630143,
1130 0x11070264, 0x1238023e, 0x1357021d, 0x14660201, 0x156601e9, 0x165a01d3, 0x174401c0, 0x182401af,
1131 0x18fe0331, 0x1a9602fe, 0x1c1502d2, 0x1d7e02ad, 0x1ed4028d, 0x201a0270, 0x21520256, 0x227d0240,
1132 0x239f0443, 0x25c003fe, 0x27bf03c4, 0x29a10392, 0x2b6a0367, 0x2d1d0341, 0x2ebe031f, 0x304d0300,
1133 0x31d105b0, 0x34a80555, 0x37520507, 0x39d504c5, 0x3c37048b, 0x3e7c0458, 0x40a8042a, 0x42bd0401,
1134 0x44c20798, 0x488e071e, 0x4c1c06b6, 0x4f76065d, 0x52a50610, 0x55ac05cc, 0x5892058f, 0x5b590559,
1135 0x5e0c0a23, 0x631c0980, 0x67db08f6, 0x6c55087f, 0x70940818, 0x74a007bd, 0x787d076c, 0x7c330723,
1136};
1137
1138static stbir__inline stbir_uint8 stbir__linear_to_srgb_uchar(float in)
1139{
1140 static const stbir__FP32 almostone = { 0x3f7fffff }; // 1-eps
1141 static const stbir__FP32 minval = { (127-13) << 23 };
1142 stbir_uint32 tab,bias,scale,t;
1143 stbir__FP32 f;
1144
1145 // Clamp to [2^(-13), 1-eps]; these two values map to 0 and 1, respectively.
1146 // The tests are carefully written so that NaNs map to 0, same as in the reference
1147 // implementation.
1148 if (!(in > minval.f)) // written this way to catch NaNs
1149 return 0;
1150 if (in > almostone.f)
1151 return 255;
1152
1153 // Do the table lookup and unpack bias, scale
1154 f.f = in;
1155 tab = fp32_to_srgb8_tab4[(f.u - minval.u) >> 20];
1156 bias = (tab >> 16) << 9;
1157 scale = tab & 0xffff;
1158
1159 // Grab next-highest mantissa bits and perform linear interpolation
1160 t = (f.u >> 12) & 0xff;
1161 return (unsigned char) ((bias + scale*t) >> 16);
1162}
1163
1164#ifndef STBIR_FORCE_GATHER_FILTER_SCANLINES_AMOUNT
1165#define STBIR_FORCE_GATHER_FILTER_SCANLINES_AMOUNT 32 // when downsampling and <= 32 scanlines of buffering, use gather. gather used down to 1/8th scaling for 25% win.
1166#endif
1167
1168#ifndef STBIR_FORCE_MINIMUM_SCANLINES_FOR_SPLITS
1169#define STBIR_FORCE_MINIMUM_SCANLINES_FOR_SPLITS 4 // when threading, what is the minimum number of scanlines for a split?
1170#endif
1171
1172#define STBIR_INPUT_CALLBACK_PADDING 3
1173
1174#ifdef _M_IX86_FP
1175#if ( _M_IX86_FP >= 1 )
1176#ifndef STBIR_SSE
1177#define STBIR_SSE
1178#endif
1179#endif
1180#endif
1181
1182#ifdef __TINYC__
1183 // tiny c has no intrinsics yet - this can become a version check if they add them
1184 #define STBIR_NO_SIMD
1185#endif
1186
1187#if defined(_x86_64) || defined( __x86_64__ ) || defined( _M_X64 ) || defined(__x86_64) || defined(_M_AMD64) || defined(__SSE2__) || defined(STBIR_SSE) || defined(STBIR_SSE2)
1188 #ifndef STBIR_SSE2
1189 #define STBIR_SSE2
1190 #endif
1191 #if defined(__AVX__) || defined(STBIR_AVX2)
1192 #ifndef STBIR_AVX
1193 #ifndef STBIR_NO_AVX
1194 #define STBIR_AVX
1195 #endif
1196 #endif
1197 #endif
1198 #if defined(__AVX2__) || defined(STBIR_AVX2)
1199 #ifndef STBIR_NO_AVX2
1200 #ifndef STBIR_AVX2
1201 #define STBIR_AVX2
1202 #endif
1203 #if defined( _MSC_VER ) && !defined(__clang__)
1204 #ifndef STBIR_FP16C // FP16C instructions are on all AVX2 cpus, so we can autoselect it here on microsoft - clang needs -mf16c
1205 #define STBIR_FP16C
1206 #endif
1207 #endif
1208 #endif
1209 #endif
1210 #ifdef __F16C__
1211 #ifndef STBIR_FP16C // turn on FP16C instructions if the define is set (for clang and gcc)
1212 #define STBIR_FP16C
1213 #endif
1214 #endif
1215#endif
1216
1217#if defined( _M_ARM64 ) || defined( __aarch64__ ) || defined( __arm64__ ) || ((__ARM_NEON_FP & 4) != 0) || defined(__ARM_NEON__)
1218#ifndef STBIR_NEON
1219#define STBIR_NEON
1220#endif
1221#endif
1222
1223#if defined(_M_ARM) || defined(__arm__)
1224#ifdef STBIR_USE_FMA
1225#undef STBIR_USE_FMA // no FMA for 32-bit arm on MSVC
1226#endif
1227#endif
1228
1229#if defined(__wasm__) && defined(__wasm_simd128__)
1230#ifndef STBIR_WASM
1231#define STBIR_WASM
1232#endif
1233#endif
1234
1235// restrict pointers for the output pointers, other loop and unroll control
1236#if defined( _MSC_VER ) && !defined(__clang__)
1237 #define STBIR_STREAMOUT_PTR( star ) star __restrict
1238 #define STBIR_NO_UNROLL( ptr ) __assume(ptr) // this oddly keeps msvc from unrolling a loop
1239 #if _MSC_VER >= 1900
1240 #define STBIR_NO_UNROLL_LOOP_START __pragma(loop( no_vector ))
1241 #else
1242 #define STBIR_NO_UNROLL_LOOP_START
1243 #endif
1244#elif defined( __clang__ )
1245 #define STBIR_STREAMOUT_PTR( star ) star __restrict__
1246 #define STBIR_NO_UNROLL( ptr ) __asm__ (""::"r"(ptr))
1247 #if ( __clang_major__ >= 4 ) || ( ( __clang_major__ >= 3 ) && ( __clang_minor__ >= 5 ) )
1248 #define STBIR_NO_UNROLL_LOOP_START _Pragma("clang loop unroll(disable)") _Pragma("clang loop vectorize(disable)")
1249 #else
1250 #define STBIR_NO_UNROLL_LOOP_START
1251 #endif
1252#elif defined( __GNUC__ )
1253 #define STBIR_STREAMOUT_PTR( star ) star __restrict__
1254 #define STBIR_NO_UNROLL( ptr ) __asm__ (""::"r"(ptr))
1255 #if __GNUC__ >= 14
1256 #define STBIR_NO_UNROLL_LOOP_START _Pragma("GCC unroll 0") _Pragma("GCC novector")
1257 #else
1258 #define STBIR_NO_UNROLL_LOOP_START
1259 #endif
1260 #define STBIR_NO_UNROLL_LOOP_START_INF_FOR
1261#else
1262 #define STBIR_STREAMOUT_PTR( star ) star
1263 #define STBIR_NO_UNROLL( ptr )
1264 #define STBIR_NO_UNROLL_LOOP_START
1265#endif
1266
1267#ifndef STBIR_NO_UNROLL_LOOP_START_INF_FOR
1268#define STBIR_NO_UNROLL_LOOP_START_INF_FOR STBIR_NO_UNROLL_LOOP_START
1269#endif
1270
1271#ifdef STBIR_NO_SIMD // force simd off for whatever reason
1272
1273// force simd off overrides everything else, so clear it all
1274
1275#ifdef STBIR_SSE2
1276#undef STBIR_SSE2
1277#endif
1278
1279#ifdef STBIR_AVX
1280#undef STBIR_AVX
1281#endif
1282
1283#ifdef STBIR_NEON
1284#undef STBIR_NEON
1285#endif
1286
1287#ifdef STBIR_AVX2
1288#undef STBIR_AVX2
1289#endif
1290
1291#ifdef STBIR_FP16C
1292#undef STBIR_FP16C
1293#endif
1294
1295#ifdef STBIR_WASM
1296#undef STBIR_WASM
1297#endif
1298
1299#ifdef STBIR_SIMD
1300#undef STBIR_SIMD
1301#endif
1302
1303#else // STBIR_SIMD
1304
1305#ifdef STBIR_SSE2
1306 #include <emmintrin.h>
1307
1308 #define stbir__simdf __m128
1309 #define stbir__simdi __m128i
1310
1311 #define stbir_simdi_castf( reg ) _mm_castps_si128(reg)
1312 #define stbir_simdf_casti( reg ) _mm_castsi128_ps(reg)
1313
1314 #define stbir__simdf_load( reg, ptr ) (reg) = _mm_loadu_ps( (float const*)(ptr) )
1315 #define stbir__simdi_load( reg, ptr ) (reg) = _mm_loadu_si128 ( (stbir__simdi const*)(ptr) )
1316 #define stbir__simdf_load1( out, ptr ) (out) = _mm_load_ss( (float const*)(ptr) ) // top values can be random (not denormal or nan for perf)
1317 #define stbir__simdi_load1( out, ptr ) (out) = _mm_castps_si128( _mm_load_ss( (float const*)(ptr) ))
1318 #define stbir__simdf_load1z( out, ptr ) (out) = _mm_load_ss( (float const*)(ptr) ) // top values must be zero
1319 #define stbir__simdf_frep4( fvar ) _mm_set_ps1( fvar )
1320 #define stbir__simdf_load1frep4( out, fvar ) (out) = _mm_set_ps1( fvar )
1321 #define stbir__simdf_load2( out, ptr ) (out) = _mm_castsi128_ps( _mm_loadl_epi64( (__m128i*)(ptr)) ) // top values can be random (not denormal or nan for perf)
1322 #define stbir__simdf_load2z( out, ptr ) (out) = _mm_castsi128_ps( _mm_loadl_epi64( (__m128i*)(ptr)) ) // top values must be zero
1323 #define stbir__simdf_load2hmerge( out, reg, ptr ) (out) = _mm_castpd_ps(_mm_loadh_pd( _mm_castps_pd(reg), (double*)(ptr) ))
1324
1325 #define stbir__simdf_zeroP() _mm_setzero_ps()
1326 #define stbir__simdf_zero( reg ) (reg) = _mm_setzero_ps()
1327
1328 #define stbir__simdf_store( ptr, reg ) _mm_storeu_ps( (float*)(ptr), reg )
1329 #define stbir__simdf_store1( ptr, reg ) _mm_store_ss( (float*)(ptr), reg )
1330 #define stbir__simdf_store2( ptr, reg ) _mm_storel_epi64( (__m128i*)(ptr), _mm_castps_si128(reg) )
1331 #define stbir__simdf_store2h( ptr, reg ) _mm_storeh_pd( (double*)(ptr), _mm_castps_pd(reg) )
1332
1333 #define stbir__simdi_store( ptr, reg ) _mm_storeu_si128( (__m128i*)(ptr), reg )
1334 #define stbir__simdi_store1( ptr, reg ) _mm_store_ss( (float*)(ptr), _mm_castsi128_ps(reg) )
1335 #define stbir__simdi_store2( ptr, reg ) _mm_storel_epi64( (__m128i*)(ptr), (reg) )
1336
1337 #define stbir__prefetch( ptr ) _mm_prefetch((char*)(ptr), _MM_HINT_T0 )
1338
1339 #define stbir__simdi_expand_u8_to_u32(out0,out1,out2,out3,ireg) \
1340 { \
1341 stbir__simdi zero = _mm_setzero_si128(); \
1342 out2 = _mm_unpacklo_epi8( ireg, zero ); \
1343 out3 = _mm_unpackhi_epi8( ireg, zero ); \
1344 out0 = _mm_unpacklo_epi16( out2, zero ); \
1345 out1 = _mm_unpackhi_epi16( out2, zero ); \
1346 out2 = _mm_unpacklo_epi16( out3, zero ); \
1347 out3 = _mm_unpackhi_epi16( out3, zero ); \
1348 }
1349
1350#define stbir__simdi_expand_u8_to_1u32(out,ireg) \
1351 { \
1352 stbir__simdi zero = _mm_setzero_si128(); \
1353 out = _mm_unpacklo_epi8( ireg, zero ); \
1354 out = _mm_unpacklo_epi16( out, zero ); \
1355 }
1356
1357 #define stbir__simdi_expand_u16_to_u32(out0,out1,ireg) \
1358 { \
1359 stbir__simdi zero = _mm_setzero_si128(); \
1360 out0 = _mm_unpacklo_epi16( ireg, zero ); \
1361 out1 = _mm_unpackhi_epi16( ireg, zero ); \
1362 }
1363
1364 #define stbir__simdf_convert_float_to_i32( i, f ) (i) = _mm_cvttps_epi32(f)
1365 #define stbir__simdf_convert_float_to_int( f ) _mm_cvtt_ss2si(f)
1366 #define stbir__simdf_convert_float_to_uint8( f ) ((unsigned char)_mm_cvtsi128_si32(_mm_cvttps_epi32(_mm_max_ps(_mm_min_ps(f,STBIR__CONSTF(STBIR_max_uint8_as_float)),_mm_setzero_ps()))))
1367 #define stbir__simdf_convert_float_to_short( f ) ((unsigned short)_mm_cvtsi128_si32(_mm_cvttps_epi32(_mm_max_ps(_mm_min_ps(f,STBIR__CONSTF(STBIR_max_uint16_as_float)),_mm_setzero_ps()))))
1368
1369 #define stbir__simdi_to_int( i ) _mm_cvtsi128_si32(i)
1370 #define stbir__simdi_convert_i32_to_float(out, ireg) (out) = _mm_cvtepi32_ps( ireg )
1371 #define stbir__simdf_add( out, reg0, reg1 ) (out) = _mm_add_ps( reg0, reg1 )
1372 #define stbir__simdf_mult( out, reg0, reg1 ) (out) = _mm_mul_ps( reg0, reg1 )
1373 #define stbir__simdf_mult_mem( out, reg, ptr ) (out) = _mm_mul_ps( reg, _mm_loadu_ps( (float const*)(ptr) ) )
1374 #define stbir__simdf_mult1_mem( out, reg, ptr ) (out) = _mm_mul_ss( reg, _mm_load_ss( (float const*)(ptr) ) )
1375 #define stbir__simdf_add_mem( out, reg, ptr ) (out) = _mm_add_ps( reg, _mm_loadu_ps( (float const*)(ptr) ) )
1376 #define stbir__simdf_add1_mem( out, reg, ptr ) (out) = _mm_add_ss( reg, _mm_load_ss( (float const*)(ptr) ) )
1377
1378 #ifdef STBIR_USE_FMA // not on by default to maintain bit identical simd to non-simd
1379 #include <immintrin.h>
1380 #define stbir__simdf_madd( out, add, mul1, mul2 ) (out) = _mm_fmadd_ps( mul1, mul2, add )
1381 #define stbir__simdf_madd1( out, add, mul1, mul2 ) (out) = _mm_fmadd_ss( mul1, mul2, add )
1382 #define stbir__simdf_madd_mem( out, add, mul, ptr ) (out) = _mm_fmadd_ps( mul, _mm_loadu_ps( (float const*)(ptr) ), add )
1383 #define stbir__simdf_madd1_mem( out, add, mul, ptr ) (out) = _mm_fmadd_ss( mul, _mm_load_ss( (float const*)(ptr) ), add )
1384 #else
1385 #define stbir__simdf_madd( out, add, mul1, mul2 ) (out) = _mm_add_ps( add, _mm_mul_ps( mul1, mul2 ) )
1386 #define stbir__simdf_madd1( out, add, mul1, mul2 ) (out) = _mm_add_ss( add, _mm_mul_ss( mul1, mul2 ) )
1387 #define stbir__simdf_madd_mem( out, add, mul, ptr ) (out) = _mm_add_ps( add, _mm_mul_ps( mul, _mm_loadu_ps( (float const*)(ptr) ) ) )
1388 #define stbir__simdf_madd1_mem( out, add, mul, ptr ) (out) = _mm_add_ss( add, _mm_mul_ss( mul, _mm_load_ss( (float const*)(ptr) ) ) )
1389 #endif
1390
1391 #define stbir__simdf_add1( out, reg0, reg1 ) (out) = _mm_add_ss( reg0, reg1 )
1392 #define stbir__simdf_mult1( out, reg0, reg1 ) (out) = _mm_mul_ss( reg0, reg1 )
1393
1394 #define stbir__simdf_and( out, reg0, reg1 ) (out) = _mm_and_ps( reg0, reg1 )
1395 #define stbir__simdf_or( out, reg0, reg1 ) (out) = _mm_or_ps( reg0, reg1 )
1396
1397 #define stbir__simdf_min( out, reg0, reg1 ) (out) = _mm_min_ps( reg0, reg1 )
1398 #define stbir__simdf_max( out, reg0, reg1 ) (out) = _mm_max_ps( reg0, reg1 )
1399 #define stbir__simdf_min1( out, reg0, reg1 ) (out) = _mm_min_ss( reg0, reg1 )
1400 #define stbir__simdf_max1( out, reg0, reg1 ) (out) = _mm_max_ss( reg0, reg1 )
1401
1402 #define stbir__simdf_0123ABCDto3ABx( out, reg0, reg1 ) (out)=_mm_castsi128_ps( _mm_shuffle_epi32( _mm_castps_si128( _mm_shuffle_ps( reg1,reg0, (0<<0) + (1<<2) + (2<<4) + (3<<6) )), (3<<0) + (0<<2) + (1<<4) + (2<<6) ) )
1403 #define stbir__simdf_0123ABCDto23Ax( out, reg0, reg1 ) (out)=_mm_castsi128_ps( _mm_shuffle_epi32( _mm_castps_si128( _mm_shuffle_ps( reg1,reg0, (0<<0) + (1<<2) + (2<<4) + (3<<6) )), (2<<0) + (3<<2) + (0<<4) + (1<<6) ) )
1404
1405 static const stbir__simdf STBIR_zeroones = { 0.0f,1.0f,0.0f,1.0f };
1406 static const stbir__simdf STBIR_onezeros = { 1.0f,0.0f,1.0f,0.0f };
1407 #define stbir__simdf_aaa1( out, alp, ones ) (out)=_mm_castsi128_ps( _mm_shuffle_epi32( _mm_castps_si128( _mm_movehl_ps( ones, alp ) ), (1<<0) + (1<<2) + (1<<4) + (2<<6) ) )
1408 #define stbir__simdf_1aaa( out, alp, ones ) (out)=_mm_castsi128_ps( _mm_shuffle_epi32( _mm_castps_si128( _mm_movelh_ps( ones, alp ) ), (0<<0) + (2<<2) + (2<<4) + (2<<6) ) )
1409 #define stbir__simdf_a1a1( out, alp, ones) (out) = _mm_or_ps( _mm_castsi128_ps( _mm_srli_epi64( _mm_castps_si128(alp), 32 ) ), STBIR_zeroones )
1410 #define stbir__simdf_1a1a( out, alp, ones) (out) = _mm_or_ps( _mm_castsi128_ps( _mm_slli_epi64( _mm_castps_si128(alp), 32 ) ), STBIR_onezeros )
1411
1412 #define stbir__simdf_swiz( reg, one, two, three, four ) _mm_castsi128_ps( _mm_shuffle_epi32( _mm_castps_si128( reg ), (one<<0) + (two<<2) + (three<<4) + (four<<6) ) )
1413
1414 #define stbir__simdi_and( out, reg0, reg1 ) (out) = _mm_and_si128( reg0, reg1 )
1415 #define stbir__simdi_or( out, reg0, reg1 ) (out) = _mm_or_si128( reg0, reg1 )
1416 #define stbir__simdi_16madd( out, reg0, reg1 ) (out) = _mm_madd_epi16( reg0, reg1 )
1417
1418 #define stbir__simdf_pack_to_8bytes(out,aa,bb) \
1419 { \
1420 stbir__simdf af,bf; \
1421 stbir__simdi a,b; \
1422 af = _mm_min_ps( aa, STBIR_max_uint8_as_float ); \
1423 bf = _mm_min_ps( bb, STBIR_max_uint8_as_float ); \
1424 af = _mm_max_ps( af, _mm_setzero_ps() ); \
1425 bf = _mm_max_ps( bf, _mm_setzero_ps() ); \
1426 a = _mm_cvttps_epi32( af ); \
1427 b = _mm_cvttps_epi32( bf ); \
1428 a = _mm_packs_epi32( a, b ); \
1429 out = _mm_packus_epi16( a, a ); \
1430 }
1431
1432 #define stbir__simdf_load4_transposed( o0, o1, o2, o3, ptr ) \
1433 stbir__simdf_load( o0, (ptr) ); \
1434 stbir__simdf_load( o1, (ptr)+4 ); \
1435 stbir__simdf_load( o2, (ptr)+8 ); \
1436 stbir__simdf_load( o3, (ptr)+12 ); \
1437 { \
1438 __m128 tmp0, tmp1, tmp2, tmp3; \
1439 tmp0 = _mm_unpacklo_ps(o0, o1); \
1440 tmp2 = _mm_unpacklo_ps(o2, o3); \
1441 tmp1 = _mm_unpackhi_ps(o0, o1); \
1442 tmp3 = _mm_unpackhi_ps(o2, o3); \
1443 o0 = _mm_movelh_ps(tmp0, tmp2); \
1444 o1 = _mm_movehl_ps(tmp2, tmp0); \
1445 o2 = _mm_movelh_ps(tmp1, tmp3); \
1446 o3 = _mm_movehl_ps(tmp3, tmp1); \
1447 }
1448
1449 #define stbir__interleave_pack_and_store_16_u8( ptr, r0, r1, r2, r3 ) \
1450 r0 = _mm_packs_epi32( r0, r1 ); \
1451 r2 = _mm_packs_epi32( r2, r3 ); \
1452 r1 = _mm_unpacklo_epi16( r0, r2 ); \
1453 r3 = _mm_unpackhi_epi16( r0, r2 ); \
1454 r0 = _mm_unpacklo_epi16( r1, r3 ); \
1455 r2 = _mm_unpackhi_epi16( r1, r3 ); \
1456 r0 = _mm_packus_epi16( r0, r2 ); \
1457 stbir__simdi_store( ptr, r0 ); \
1458
1459 #define stbir__simdi_32shr( out, reg, imm ) out = _mm_srli_epi32( reg, imm )
1460
1461 #if defined(_MSC_VER) && !defined(__clang__)
1462 // msvc inits with 8 bytes
1463 #define STBIR__CONST_32_TO_8( v ) (char)(unsigned char)((v)&255),(char)(unsigned char)(((v)>>8)&255),(char)(unsigned char)(((v)>>16)&255),(char)(unsigned char)(((v)>>24)&255)
1464 #define STBIR__CONST_4_32i( v ) STBIR__CONST_32_TO_8( v ), STBIR__CONST_32_TO_8( v ), STBIR__CONST_32_TO_8( v ), STBIR__CONST_32_TO_8( v )
1465 #define STBIR__CONST_4d_32i( v0, v1, v2, v3 ) STBIR__CONST_32_TO_8( v0 ), STBIR__CONST_32_TO_8( v1 ), STBIR__CONST_32_TO_8( v2 ), STBIR__CONST_32_TO_8( v3 )
1466 #else
1467 // everything else inits with long long's
1468 #define STBIR__CONST_4_32i( v ) (long long)((((stbir_uint64)(stbir_uint32)(v))<<32)|((stbir_uint64)(stbir_uint32)(v))),(long long)((((stbir_uint64)(stbir_uint32)(v))<<32)|((stbir_uint64)(stbir_uint32)(v)))
1469 #define STBIR__CONST_4d_32i( v0, v1, v2, v3 ) (long long)((((stbir_uint64)(stbir_uint32)(v1))<<32)|((stbir_uint64)(stbir_uint32)(v0))),(long long)((((stbir_uint64)(stbir_uint32)(v3))<<32)|((stbir_uint64)(stbir_uint32)(v2)))
1470 #endif
1471
1472 #define STBIR__SIMDF_CONST(var, x) stbir__simdf var = { x, x, x, x }
1473 #define STBIR__SIMDI_CONST(var, x) stbir__simdi var = { STBIR__CONST_4_32i(x) }
1474 #define STBIR__CONSTF(var) (var)
1475 #define STBIR__CONSTI(var) (var)
1476
1477 #if defined(STBIR_AVX) || defined(__SSE4_1__)
1478 #include <smmintrin.h>
1479 #define stbir__simdf_pack_to_8words(out,reg0,reg1) out = _mm_packus_epi32(_mm_cvttps_epi32(_mm_max_ps(_mm_min_ps(reg0,STBIR__CONSTF(STBIR_max_uint16_as_float)),_mm_setzero_ps())), _mm_cvttps_epi32(_mm_max_ps(_mm_min_ps(reg1,STBIR__CONSTF(STBIR_max_uint16_as_float)),_mm_setzero_ps())))
1480 #else
1481 static STBIR__SIMDI_CONST(stbir__s32_32768, 32768);
1482 static STBIR__SIMDI_CONST(stbir__s16_32768, ((32768<<16)|32768));
1483
1484 #define stbir__simdf_pack_to_8words(out,reg0,reg1) \
1485 { \
1486 stbir__simdi tmp0,tmp1; \
1487 tmp0 = _mm_cvttps_epi32(_mm_max_ps(_mm_min_ps(reg0,STBIR__CONSTF(STBIR_max_uint16_as_float)),_mm_setzero_ps())); \
1488 tmp1 = _mm_cvttps_epi32(_mm_max_ps(_mm_min_ps(reg1,STBIR__CONSTF(STBIR_max_uint16_as_float)),_mm_setzero_ps())); \
1489 tmp0 = _mm_sub_epi32( tmp0, stbir__s32_32768 ); \
1490 tmp1 = _mm_sub_epi32( tmp1, stbir__s32_32768 ); \
1491 out = _mm_packs_epi32( tmp0, tmp1 ); \
1492 out = _mm_sub_epi16( out, stbir__s16_32768 ); \
1493 }
1494
1495 #endif
1496
1497 #define STBIR_SIMD
1498
1499 // if we detect AVX, set the simd8 defines
1500 #ifdef STBIR_AVX
1501 #include <immintrin.h>
1502 #define STBIR_SIMD8
1503 #define stbir__simdf8 __m256
1504 #define stbir__simdi8 __m256i
1505 #define stbir__simdf8_load( out, ptr ) (out) = _mm256_loadu_ps( (float const *)(ptr) )
1506 #define stbir__simdi8_load( out, ptr ) (out) = _mm256_loadu_si256( (__m256i const *)(ptr) )
1507 #define stbir__simdf8_mult( out, a, b ) (out) = _mm256_mul_ps( (a), (b) )
1508 #define stbir__simdf8_store( ptr, out ) _mm256_storeu_ps( (float*)(ptr), out )
1509 #define stbir__simdi8_store( ptr, reg ) _mm256_storeu_si256( (__m256i*)(ptr), reg )
1510 #define stbir__simdf8_frep8( fval ) _mm256_set1_ps( fval )
1511
1512 #define stbir__simdf8_min( out, reg0, reg1 ) (out) = _mm256_min_ps( reg0, reg1 )
1513 #define stbir__simdf8_max( out, reg0, reg1 ) (out) = _mm256_max_ps( reg0, reg1 )
1514
1515 #define stbir__simdf8_add4halves( out, bot4, top8 ) (out) = _mm_add_ps( bot4, _mm256_extractf128_ps( top8, 1 ) )
1516 #define stbir__simdf8_mult_mem( out, reg, ptr ) (out) = _mm256_mul_ps( reg, _mm256_loadu_ps( (float const*)(ptr) ) )
1517 #define stbir__simdf8_add_mem( out, reg, ptr ) (out) = _mm256_add_ps( reg, _mm256_loadu_ps( (float const*)(ptr) ) )
1518 #define stbir__simdf8_add( out, a, b ) (out) = _mm256_add_ps( a, b )
1519 #define stbir__simdf8_load1b( out, ptr ) (out) = _mm256_broadcast_ss( ptr )
1520 #define stbir__simdf_load1rep4( out, ptr ) (out) = _mm_broadcast_ss( ptr ) // avx load instruction
1521
1522 #define stbir__simdi8_convert_i32_to_float(out, ireg) (out) = _mm256_cvtepi32_ps( ireg )
1523 #define stbir__simdf8_convert_float_to_i32( i, f ) (i) = _mm256_cvttps_epi32(f)
1524
1525 #define stbir__simdf8_bot4s( out, a, b ) (out) = _mm256_permute2f128_ps(a,b, (0<<0)+(2<<4) )
1526 #define stbir__simdf8_top4s( out, a, b ) (out) = _mm256_permute2f128_ps(a,b, (1<<0)+(3<<4) )
1527
1528 #define stbir__simdf8_gettop4( reg ) _mm256_extractf128_ps(reg,1)
1529
1530 #ifdef STBIR_AVX2
1531
1532 #define stbir__simdi8_expand_u8_to_u32(out0,out1,ireg) \
1533 { \
1534 stbir__simdi8 a, zero =_mm256_setzero_si256();\
1535 a = _mm256_permute4x64_epi64( _mm256_unpacklo_epi8( _mm256_permute4x64_epi64(_mm256_castsi128_si256(ireg),(0<<0)+(2<<2)+(1<<4)+(3<<6)), zero ),(0<<0)+(2<<2)+(1<<4)+(3<<6)); \
1536 out0 = _mm256_unpacklo_epi16( a, zero ); \
1537 out1 = _mm256_unpackhi_epi16( a, zero ); \
1538 }
1539
1540 #define stbir__simdf8_pack_to_16bytes(out,aa,bb) \
1541 { \
1542 stbir__simdi8 t; \
1543 stbir__simdf8 af,bf; \
1544 stbir__simdi8 a,b; \
1545 af = _mm256_min_ps( aa, STBIR_max_uint8_as_floatX ); \
1546 bf = _mm256_min_ps( bb, STBIR_max_uint8_as_floatX ); \
1547 af = _mm256_max_ps( af, _mm256_setzero_ps() ); \
1548 bf = _mm256_max_ps( bf, _mm256_setzero_ps() ); \
1549 a = _mm256_cvttps_epi32( af ); \
1550 b = _mm256_cvttps_epi32( bf ); \
1551 t = _mm256_permute4x64_epi64( _mm256_packs_epi32( a, b ), (0<<0)+(2<<2)+(1<<4)+(3<<6) ); \
1552 out = _mm256_castsi256_si128( _mm256_permute4x64_epi64( _mm256_packus_epi16( t, t ), (0<<0)+(2<<2)+(1<<4)+(3<<6) ) ); \
1553 }
1554
1555 #define stbir__simdi8_expand_u16_to_u32(out,ireg) out = _mm256_unpacklo_epi16( _mm256_permute4x64_epi64(_mm256_castsi128_si256(ireg),(0<<0)+(2<<2)+(1<<4)+(3<<6)), _mm256_setzero_si256() );
1556
1557 #define stbir__simdf8_pack_to_16words(out,aa,bb) \
1558 { \
1559 stbir__simdf8 af,bf; \
1560 stbir__simdi8 a,b; \
1561 af = _mm256_min_ps( aa, STBIR_max_uint16_as_floatX ); \
1562 bf = _mm256_min_ps( bb, STBIR_max_uint16_as_floatX ); \
1563 af = _mm256_max_ps( af, _mm256_setzero_ps() ); \
1564 bf = _mm256_max_ps( bf, _mm256_setzero_ps() ); \
1565 a = _mm256_cvttps_epi32( af ); \
1566 b = _mm256_cvttps_epi32( bf ); \
1567 (out) = _mm256_permute4x64_epi64( _mm256_packus_epi32(a, b), (0<<0)+(2<<2)+(1<<4)+(3<<6) ); \
1568 }
1569
1570 #else
1571
1572 #define stbir__simdi8_expand_u8_to_u32(out0,out1,ireg) \
1573 { \
1574 stbir__simdi a,zero = _mm_setzero_si128(); \
1575 a = _mm_unpacklo_epi8( ireg, zero ); \
1576 out0 = _mm256_setr_m128i( _mm_unpacklo_epi16( a, zero ), _mm_unpackhi_epi16( a, zero ) ); \
1577 a = _mm_unpackhi_epi8( ireg, zero ); \
1578 out1 = _mm256_setr_m128i( _mm_unpacklo_epi16( a, zero ), _mm_unpackhi_epi16( a, zero ) ); \
1579 }
1580
1581 #define stbir__simdf8_pack_to_16bytes(out,aa,bb) \
1582 { \
1583 stbir__simdi t; \
1584 stbir__simdf8 af,bf; \
1585 stbir__simdi8 a,b; \
1586 af = _mm256_min_ps( aa, STBIR_max_uint8_as_floatX ); \
1587 bf = _mm256_min_ps( bb, STBIR_max_uint8_as_floatX ); \
1588 af = _mm256_max_ps( af, _mm256_setzero_ps() ); \
1589 bf = _mm256_max_ps( bf, _mm256_setzero_ps() ); \
1590 a = _mm256_cvttps_epi32( af ); \
1591 b = _mm256_cvttps_epi32( bf ); \
1592 out = _mm_packs_epi32( _mm256_castsi256_si128(a), _mm256_extractf128_si256( a, 1 ) ); \
1593 out = _mm_packus_epi16( out, out ); \
1594 t = _mm_packs_epi32( _mm256_castsi256_si128(b), _mm256_extractf128_si256( b, 1 ) ); \
1595 t = _mm_packus_epi16( t, t ); \
1596 out = _mm_castps_si128( _mm_shuffle_ps( _mm_castsi128_ps(out), _mm_castsi128_ps(t), (0<<0)+(1<<2)+(0<<4)+(1<<6) ) ); \
1597 }
1598
1599 #define stbir__simdi8_expand_u16_to_u32(out,ireg) \
1600 { \
1601 stbir__simdi a,b,zero = _mm_setzero_si128(); \
1602 a = _mm_unpacklo_epi16( ireg, zero ); \
1603 b = _mm_unpackhi_epi16( ireg, zero ); \
1604 out = _mm256_insertf128_si256( _mm256_castsi128_si256( a ), b, 1 ); \
1605 }
1606
1607 #define stbir__simdf8_pack_to_16words(out,aa,bb) \
1608 { \
1609 stbir__simdi t0,t1; \
1610 stbir__simdf8 af,bf; \
1611 stbir__simdi8 a,b; \
1612 af = _mm256_min_ps( aa, STBIR_max_uint16_as_floatX ); \
1613 bf = _mm256_min_ps( bb, STBIR_max_uint16_as_floatX ); \
1614 af = _mm256_max_ps( af, _mm256_setzero_ps() ); \
1615 bf = _mm256_max_ps( bf, _mm256_setzero_ps() ); \
1616 a = _mm256_cvttps_epi32( af ); \
1617 b = _mm256_cvttps_epi32( bf ); \
1618 t0 = _mm_packus_epi32( _mm256_castsi256_si128(a), _mm256_extractf128_si256( a, 1 ) ); \
1619 t1 = _mm_packus_epi32( _mm256_castsi256_si128(b), _mm256_extractf128_si256( b, 1 ) ); \
1620 out = _mm256_setr_m128i( t0, t1 ); \
1621 }
1622
1623 #endif
1624
1625 static __m256i stbir_00001111 = { STBIR__CONST_4d_32i( 0, 0, 0, 0 ), STBIR__CONST_4d_32i( 1, 1, 1, 1 ) };
1626 #define stbir__simdf8_0123to00001111( out, in ) (out) = _mm256_permutevar_ps ( in, stbir_00001111 )
1627
1628 static __m256i stbir_22223333 = { STBIR__CONST_4d_32i( 2, 2, 2, 2 ), STBIR__CONST_4d_32i( 3, 3, 3, 3 ) };
1629 #define stbir__simdf8_0123to22223333( out, in ) (out) = _mm256_permutevar_ps ( in, stbir_22223333 )
1630
1631 #define stbir__simdf8_0123to2222( out, in ) (out) = stbir__simdf_swiz(_mm256_castps256_ps128(in), 2,2,2,2 )
1632
1633 #define stbir__simdf8_load4b( out, ptr ) (out) = _mm256_broadcast_ps( (__m128 const *)(ptr) )
1634
1635 static __m256i stbir_00112233 = { STBIR__CONST_4d_32i( 0, 0, 1, 1 ), STBIR__CONST_4d_32i( 2, 2, 3, 3 ) };
1636 #define stbir__simdf8_0123to00112233( out, in ) (out) = _mm256_permutevar_ps ( in, stbir_00112233 )
1637 #define stbir__simdf8_add4( out, a8, b ) (out) = _mm256_add_ps( a8, _mm256_castps128_ps256( b ) )
1638
1639 static __m256i stbir_load6 = { STBIR__CONST_4_32i( 0x80000000 ), STBIR__CONST_4d_32i( 0x80000000, 0x80000000, 0, 0 ) };
1640 #define stbir__simdf8_load6z( out, ptr ) (out) = _mm256_maskload_ps( ptr, stbir_load6 )
1641
1642 #define stbir__simdf8_0123to00000000( out, in ) (out) = _mm256_shuffle_ps ( in, in, (0<<0)+(0<<2)+(0<<4)+(0<<6) )
1643 #define stbir__simdf8_0123to11111111( out, in ) (out) = _mm256_shuffle_ps ( in, in, (1<<0)+(1<<2)+(1<<4)+(1<<6) )
1644 #define stbir__simdf8_0123to22222222( out, in ) (out) = _mm256_shuffle_ps ( in, in, (2<<0)+(2<<2)+(2<<4)+(2<<6) )
1645 #define stbir__simdf8_0123to33333333( out, in ) (out) = _mm256_shuffle_ps ( in, in, (3<<0)+(3<<2)+(3<<4)+(3<<6) )
1646 #define stbir__simdf8_0123to21032103( out, in ) (out) = _mm256_shuffle_ps ( in, in, (2<<0)+(1<<2)+(0<<4)+(3<<6) )
1647 #define stbir__simdf8_0123to32103210( out, in ) (out) = _mm256_shuffle_ps ( in, in, (3<<0)+(2<<2)+(1<<4)+(0<<6) )
1648 #define stbir__simdf8_0123to12301230( out, in ) (out) = _mm256_shuffle_ps ( in, in, (1<<0)+(2<<2)+(3<<4)+(0<<6) )
1649 #define stbir__simdf8_0123to10321032( out, in ) (out) = _mm256_shuffle_ps ( in, in, (1<<0)+(0<<2)+(3<<4)+(2<<6) )
1650 #define stbir__simdf8_0123to30123012( out, in ) (out) = _mm256_shuffle_ps ( in, in, (3<<0)+(0<<2)+(1<<4)+(2<<6) )
1651
1652 #define stbir__simdf8_0123to11331133( out, in ) (out) = _mm256_shuffle_ps ( in, in, (1<<0)+(1<<2)+(3<<4)+(3<<6) )
1653 #define stbir__simdf8_0123to00220022( out, in ) (out) = _mm256_shuffle_ps ( in, in, (0<<0)+(0<<2)+(2<<4)+(2<<6) )
1654
1655 #define stbir__simdf8_aaa1( out, alp, ones ) (out) = _mm256_blend_ps( alp, ones, (1<<0)+(1<<1)+(1<<2)+(0<<3)+(1<<4)+(1<<5)+(1<<6)+(0<<7)); (out)=_mm256_shuffle_ps( out,out, (3<<0) + (3<<2) + (3<<4) + (0<<6) )
1656 #define stbir__simdf8_1aaa( out, alp, ones ) (out) = _mm256_blend_ps( alp, ones, (0<<0)+(1<<1)+(1<<2)+(1<<3)+(0<<4)+(1<<5)+(1<<6)+(1<<7)); (out)=_mm256_shuffle_ps( out,out, (1<<0) + (0<<2) + (0<<4) + (0<<6) )
1657 #define stbir__simdf8_a1a1( out, alp, ones) (out) = _mm256_blend_ps( alp, ones, (1<<0)+(0<<1)+(1<<2)+(0<<3)+(1<<4)+(0<<5)+(1<<6)+(0<<7)); (out)=_mm256_shuffle_ps( out,out, (1<<0) + (0<<2) + (3<<4) + (2<<6) )
1658 #define stbir__simdf8_1a1a( out, alp, ones) (out) = _mm256_blend_ps( alp, ones, (0<<0)+(1<<1)+(0<<2)+(1<<3)+(0<<4)+(1<<5)+(0<<6)+(1<<7)); (out)=_mm256_shuffle_ps( out,out, (1<<0) + (0<<2) + (3<<4) + (2<<6) )
1659
1660 #define stbir__simdf8_zero( reg ) (reg) = _mm256_setzero_ps()
1661
1662 #ifdef STBIR_USE_FMA // not on by default to maintain bit identical simd to non-simd
1663 #define stbir__simdf8_madd( out, add, mul1, mul2 ) (out) = _mm256_fmadd_ps( mul1, mul2, add )
1664 #define stbir__simdf8_madd_mem( out, add, mul, ptr ) (out) = _mm256_fmadd_ps( mul, _mm256_loadu_ps( (float const*)(ptr) ), add )
1665 #define stbir__simdf8_madd_mem4( out, add, mul, ptr )(out) = _mm256_fmadd_ps( _mm256_setr_m128( mul, _mm_setzero_ps() ), _mm256_setr_m128( _mm_loadu_ps( (float const*)(ptr) ), _mm_setzero_ps() ), add )
1666 #else
1667 #define stbir__simdf8_madd( out, add, mul1, mul2 ) (out) = _mm256_add_ps( add, _mm256_mul_ps( mul1, mul2 ) )
1668 #define stbir__simdf8_madd_mem( out, add, mul, ptr ) (out) = _mm256_add_ps( add, _mm256_mul_ps( mul, _mm256_loadu_ps( (float const*)(ptr) ) ) )
1669 #define stbir__simdf8_madd_mem4( out, add, mul, ptr ) (out) = _mm256_add_ps( add, _mm256_setr_m128( _mm_mul_ps( mul, _mm_loadu_ps( (float const*)(ptr) ) ), _mm_setzero_ps() ) )
1670 #endif
1671 #define stbir__if_simdf8_cast_to_simdf4( val ) _mm256_castps256_ps128( val )
1672
1673 #endif
1674
1675 #ifdef STBIR_FLOORF
1676 #undef STBIR_FLOORF
1677 #endif
1678 #define STBIR_FLOORF stbir_simd_floorf
1679 static stbir__inline float stbir_simd_floorf(float x) // martins floorf
1680 {
1681 #if defined(STBIR_AVX) || defined(__SSE4_1__) || defined(STBIR_SSE41)
1682 __m128 t = _mm_set_ss(x);
1683 return _mm_cvtss_f32( _mm_floor_ss(t, t) );
1684 #else
1685 __m128 f = _mm_set_ss(x);
1686 __m128 t = _mm_cvtepi32_ps(_mm_cvttps_epi32(f));
1687 __m128 r = _mm_add_ss(t, _mm_and_ps(_mm_cmplt_ss(f, t), _mm_set_ss(-1.0f)));
1688 return _mm_cvtss_f32(r);
1689 #endif
1690 }
1691
1692 #ifdef STBIR_CEILF
1693 #undef STBIR_CEILF
1694 #endif
1695 #define STBIR_CEILF stbir_simd_ceilf
1696 static stbir__inline float stbir_simd_ceilf(float x) // martins ceilf
1697 {
1698 #if defined(STBIR_AVX) || defined(__SSE4_1__) || defined(STBIR_SSE41)
1699 __m128 t = _mm_set_ss(x);
1700 return _mm_cvtss_f32( _mm_ceil_ss(t, t) );
1701 #else
1702 __m128 f = _mm_set_ss(x);
1703 __m128 t = _mm_cvtepi32_ps(_mm_cvttps_epi32(f));
1704 __m128 r = _mm_add_ss(t, _mm_and_ps(_mm_cmplt_ss(t, f), _mm_set_ss(1.0f)));
1705 return _mm_cvtss_f32(r);
1706 #endif
1707 }
1708
1709#elif defined(STBIR_NEON)
1710
1711 #include <arm_neon.h>
1712
1713 #define stbir__simdf float32x4_t
1714 #define stbir__simdi uint32x4_t
1715
1716 #define stbir_simdi_castf( reg ) vreinterpretq_u32_f32(reg)
1717 #define stbir_simdf_casti( reg ) vreinterpretq_f32_u32(reg)
1718
1719 #define stbir__simdf_load( reg, ptr ) (reg) = vld1q_f32( (float const*)(ptr) )
1720 #define stbir__simdi_load( reg, ptr ) (reg) = vld1q_u32( (uint32_t const*)(ptr) )
1721 #define stbir__simdf_load1( out, ptr ) (out) = vld1q_dup_f32( (float const*)(ptr) ) // top values can be random (not denormal or nan for perf)
1722 #define stbir__simdi_load1( out, ptr ) (out) = vld1q_dup_u32( (uint32_t const*)(ptr) )
1723 #define stbir__simdf_load1z( out, ptr ) (out) = vld1q_lane_f32( (float const*)(ptr), vdupq_n_f32(0), 0 ) // top values must be zero
1724 #define stbir__simdf_frep4( fvar ) vdupq_n_f32( fvar )
1725 #define stbir__simdf_load1frep4( out, fvar ) (out) = vdupq_n_f32( fvar )
1726 #define stbir__simdf_load2( out, ptr ) (out) = vcombine_f32( vld1_f32( (float const*)(ptr) ), vcreate_f32(0) ) // top values can be random (not denormal or nan for perf)
1727 #define stbir__simdf_load2z( out, ptr ) (out) = vcombine_f32( vld1_f32( (float const*)(ptr) ), vcreate_f32(0) ) // top values must be zero
1728 #define stbir__simdf_load2hmerge( out, reg, ptr ) (out) = vcombine_f32( vget_low_f32(reg), vld1_f32( (float const*)(ptr) ) )
1729
1730 #define stbir__simdf_zeroP() vdupq_n_f32(0)
1731 #define stbir__simdf_zero( reg ) (reg) = vdupq_n_f32(0)
1732
1733 #define stbir__simdf_store( ptr, reg ) vst1q_f32( (float*)(ptr), reg )
1734 #define stbir__simdf_store1( ptr, reg ) vst1q_lane_f32( (float*)(ptr), reg, 0)
1735 #define stbir__simdf_store2( ptr, reg ) vst1_f32( (float*)(ptr), vget_low_f32(reg) )
1736 #define stbir__simdf_store2h( ptr, reg ) vst1_f32( (float*)(ptr), vget_high_f32(reg) )
1737
1738 #define stbir__simdi_store( ptr, reg ) vst1q_u32( (uint32_t*)(ptr), reg )
1739 #define stbir__simdi_store1( ptr, reg ) vst1q_lane_u32( (uint32_t*)(ptr), reg, 0 )
1740 #define stbir__simdi_store2( ptr, reg ) vst1_u32( (uint32_t*)(ptr), vget_low_u32(reg) )
1741
1742 #define stbir__prefetch( ptr )
1743
1744 #define stbir__simdi_expand_u8_to_u32(out0,out1,out2,out3,ireg) \
1745 { \
1746 uint16x8_t l = vmovl_u8( vget_low_u8 ( vreinterpretq_u8_u32(ireg) ) ); \
1747 uint16x8_t h = vmovl_u8( vget_high_u8( vreinterpretq_u8_u32(ireg) ) ); \
1748 out0 = vmovl_u16( vget_low_u16 ( l ) ); \
1749 out1 = vmovl_u16( vget_high_u16( l ) ); \
1750 out2 = vmovl_u16( vget_low_u16 ( h ) ); \
1751 out3 = vmovl_u16( vget_high_u16( h ) ); \
1752 }
1753
1754 #define stbir__simdi_expand_u8_to_1u32(out,ireg) \
1755 { \
1756 uint16x8_t tmp = vmovl_u8( vget_low_u8( vreinterpretq_u8_u32(ireg) ) ); \
1757 out = vmovl_u16( vget_low_u16( tmp ) ); \
1758 }
1759
1760 #define stbir__simdi_expand_u16_to_u32(out0,out1,ireg) \
1761 { \
1762 uint16x8_t tmp = vreinterpretq_u16_u32(ireg); \
1763 out0 = vmovl_u16( vget_low_u16 ( tmp ) ); \
1764 out1 = vmovl_u16( vget_high_u16( tmp ) ); \
1765 }
1766
1767 #define stbir__simdf_convert_float_to_i32( i, f ) (i) = vreinterpretq_u32_s32( vcvtq_s32_f32(f) )
1768 #define stbir__simdf_convert_float_to_int( f ) vgetq_lane_s32(vcvtq_s32_f32(f), 0)
1769 #define stbir__simdi_to_int( i ) (int)vgetq_lane_u32(i, 0)
1770 #define stbir__simdf_convert_float_to_uint8( f ) ((unsigned char)vgetq_lane_s32(vcvtq_s32_f32(vmaxq_f32(vminq_f32(f,STBIR__CONSTF(STBIR_max_uint8_as_float)),vdupq_n_f32(0))), 0))
1771 #define stbir__simdf_convert_float_to_short( f ) ((unsigned short)vgetq_lane_s32(vcvtq_s32_f32(vmaxq_f32(vminq_f32(f,STBIR__CONSTF(STBIR_max_uint16_as_float)),vdupq_n_f32(0))), 0))
1772 #define stbir__simdi_convert_i32_to_float(out, ireg) (out) = vcvtq_f32_s32( vreinterpretq_s32_u32(ireg) )
1773 #define stbir__simdf_add( out, reg0, reg1 ) (out) = vaddq_f32( reg0, reg1 )
1774 #define stbir__simdf_mult( out, reg0, reg1 ) (out) = vmulq_f32( reg0, reg1 )
1775 #define stbir__simdf_mult_mem( out, reg, ptr ) (out) = vmulq_f32( reg, vld1q_f32( (float const*)(ptr) ) )
1776 #define stbir__simdf_mult1_mem( out, reg, ptr ) (out) = vmulq_f32( reg, vld1q_dup_f32( (float const*)(ptr) ) )
1777 #define stbir__simdf_add_mem( out, reg, ptr ) (out) = vaddq_f32( reg, vld1q_f32( (float const*)(ptr) ) )
1778 #define stbir__simdf_add1_mem( out, reg, ptr ) (out) = vaddq_f32( reg, vld1q_dup_f32( (float const*)(ptr) ) )
1779
1780 #ifdef STBIR_USE_FMA // not on by default to maintain bit identical simd to non-simd (and also x64 no madd to arm madd)
1781 #define stbir__simdf_madd( out, add, mul1, mul2 ) (out) = vfmaq_f32( add, mul1, mul2 )
1782 #define stbir__simdf_madd1( out, add, mul1, mul2 ) (out) = vfmaq_f32( add, mul1, mul2 )
1783 #define stbir__simdf_madd_mem( out, add, mul, ptr ) (out) = vfmaq_f32( add, mul, vld1q_f32( (float const*)(ptr) ) )
1784 #define stbir__simdf_madd1_mem( out, add, mul, ptr ) (out) = vfmaq_f32( add, mul, vld1q_dup_f32( (float const*)(ptr) ) )
1785 #else
1786 #define stbir__simdf_madd( out, add, mul1, mul2 ) (out) = vaddq_f32( add, vmulq_f32( mul1, mul2 ) )
1787 #define stbir__simdf_madd1( out, add, mul1, mul2 ) (out) = vaddq_f32( add, vmulq_f32( mul1, mul2 ) )
1788 #define stbir__simdf_madd_mem( out, add, mul, ptr ) (out) = vaddq_f32( add, vmulq_f32( mul, vld1q_f32( (float const*)(ptr) ) ) )
1789 #define stbir__simdf_madd1_mem( out, add, mul, ptr ) (out) = vaddq_f32( add, vmulq_f32( mul, vld1q_dup_f32( (float const*)(ptr) ) ) )
1790 #endif
1791
1792 #define stbir__simdf_add1( out, reg0, reg1 ) (out) = vaddq_f32( reg0, reg1 )
1793 #define stbir__simdf_mult1( out, reg0, reg1 ) (out) = vmulq_f32( reg0, reg1 )
1794
1795 #define stbir__simdf_and( out, reg0, reg1 ) (out) = vreinterpretq_f32_u32( vandq_u32( vreinterpretq_u32_f32(reg0), vreinterpretq_u32_f32(reg1) ) )
1796 #define stbir__simdf_or( out, reg0, reg1 ) (out) = vreinterpretq_f32_u32( vorrq_u32( vreinterpretq_u32_f32(reg0), vreinterpretq_u32_f32(reg1) ) )
1797
1798 #define stbir__simdf_min( out, reg0, reg1 ) (out) = vminq_f32( reg0, reg1 )
1799 #define stbir__simdf_max( out, reg0, reg1 ) (out) = vmaxq_f32( reg0, reg1 )
1800 #define stbir__simdf_min1( out, reg0, reg1 ) (out) = vminq_f32( reg0, reg1 )
1801 #define stbir__simdf_max1( out, reg0, reg1 ) (out) = vmaxq_f32( reg0, reg1 )
1802
1803 #define stbir__simdf_0123ABCDto3ABx( out, reg0, reg1 ) (out) = vextq_f32( reg0, reg1, 3 )
1804 #define stbir__simdf_0123ABCDto23Ax( out, reg0, reg1 ) (out) = vextq_f32( reg0, reg1, 2 )
1805
1806 #define stbir__simdf_a1a1( out, alp, ones ) (out) = vzipq_f32(vuzpq_f32(alp, alp).val[1], ones).val[0]
1807 #define stbir__simdf_1a1a( out, alp, ones ) (out) = vzipq_f32(ones, vuzpq_f32(alp, alp).val[0]).val[0]
1808
1809 #if defined( _M_ARM64 ) || defined( __aarch64__ ) || defined( __arm64__ )
1810
1811 #define stbir__simdf_aaa1( out, alp, ones ) (out) = vcopyq_laneq_f32(vdupq_n_f32(vgetq_lane_f32(alp, 3)), 3, ones, 3)
1812 #define stbir__simdf_1aaa( out, alp, ones ) (out) = vcopyq_laneq_f32(vdupq_n_f32(vgetq_lane_f32(alp, 0)), 0, ones, 0)
1813
1814 #if defined( _MSC_VER ) && !defined(__clang__)
1815 #define stbir_make16(a,b,c,d) vcombine_u8( \
1816 vcreate_u8( (4*a+0) | ((4*a+1)<<8) | ((4*a+2)<<16) | ((4*a+3)<<24) | \
1817 ((stbir_uint64)(4*b+0)<<32) | ((stbir_uint64)(4*b+1)<<40) | ((stbir_uint64)(4*b+2)<<48) | ((stbir_uint64)(4*b+3)<<56)), \
1818 vcreate_u8( (4*c+0) | ((4*c+1)<<8) | ((4*c+2)<<16) | ((4*c+3)<<24) | \
1819 ((stbir_uint64)(4*d+0)<<32) | ((stbir_uint64)(4*d+1)<<40) | ((stbir_uint64)(4*d+2)<<48) | ((stbir_uint64)(4*d+3)<<56) ) )
1820
1821 static stbir__inline uint8x16x2_t stbir_make16x2(float32x4_t rega,float32x4_t regb)
1822 {
1823 uint8x16x2_t r = { vreinterpretq_u8_f32(rega), vreinterpretq_u8_f32(regb) };
1824 return r;
1825 }
1826 #else
1827 #define stbir_make16(a,b,c,d) (uint8x16_t){4*a+0,4*a+1,4*a+2,4*a+3,4*b+0,4*b+1,4*b+2,4*b+3,4*c+0,4*c+1,4*c+2,4*c+3,4*d+0,4*d+1,4*d+2,4*d+3}
1828 #define stbir_make16x2(a,b) (uint8x16x2_t){{vreinterpretq_u8_f32(a),vreinterpretq_u8_f32(b)}}
1829 #endif
1830
1831 #define stbir__simdf_swiz( reg, one, two, three, four ) vreinterpretq_f32_u8( vqtbl1q_u8( vreinterpretq_u8_f32(reg), stbir_make16(one, two, three, four) ) )
1832 #define stbir__simdf_swiz2( rega, regb, one, two, three, four ) vreinterpretq_f32_u8( vqtbl2q_u8( stbir_make16x2(rega,regb), stbir_make16(one, two, three, four) ) )
1833
1834 #define stbir__simdi_16madd( out, reg0, reg1 ) \
1835 { \
1836 int16x8_t r0 = vreinterpretq_s16_u32(reg0); \
1837 int16x8_t r1 = vreinterpretq_s16_u32(reg1); \
1838 int32x4_t tmp0 = vmull_s16( vget_low_s16(r0), vget_low_s16(r1) ); \
1839 int32x4_t tmp1 = vmull_s16( vget_high_s16(r0), vget_high_s16(r1) ); \
1840 (out) = vreinterpretq_u32_s32( vpaddq_s32(tmp0, tmp1) ); \
1841 }
1842
1843 #else
1844
1845 #define stbir__simdf_aaa1( out, alp, ones ) (out) = vsetq_lane_f32(1.0f, vdupq_n_f32(vgetq_lane_f32(alp, 3)), 3)
1846 #define stbir__simdf_1aaa( out, alp, ones ) (out) = vsetq_lane_f32(1.0f, vdupq_n_f32(vgetq_lane_f32(alp, 0)), 0)
1847
1848 #if defined( _MSC_VER ) && !defined(__clang__)
1849 static stbir__inline uint8x8x2_t stbir_make8x2(float32x4_t reg)
1850 {
1851 uint8x8x2_t r = { { vget_low_u8(vreinterpretq_u8_f32(reg)), vget_high_u8(vreinterpretq_u8_f32(reg)) } };
1852 return r;
1853 }
1854 #define stbir_make8(a,b) vcreate_u8( \
1855 (4*a+0) | ((4*a+1)<<8) | ((4*a+2)<<16) | ((4*a+3)<<24) | \
1856 ((stbir_uint64)(4*b+0)<<32) | ((stbir_uint64)(4*b+1)<<40) | ((stbir_uint64)(4*b+2)<<48) | ((stbir_uint64)(4*b+3)<<56) )
1857 #else
1858 #define stbir_make8x2(reg) (uint8x8x2_t){ { vget_low_u8(vreinterpretq_u8_f32(reg)), vget_high_u8(vreinterpretq_u8_f32(reg)) } }
1859 #define stbir_make8(a,b) (uint8x8_t){4*a+0,4*a+1,4*a+2,4*a+3,4*b+0,4*b+1,4*b+2,4*b+3}
1860 #endif
1861
1862 #define stbir__simdf_swiz( reg, one, two, three, four ) vreinterpretq_f32_u8( vcombine_u8( \
1863 vtbl2_u8( stbir_make8x2( reg ), stbir_make8( one, two ) ), \
1864 vtbl2_u8( stbir_make8x2( reg ), stbir_make8( three, four ) ) ) )
1865
1866 #define stbir__simdi_16madd( out, reg0, reg1 ) \
1867 { \
1868 int16x8_t r0 = vreinterpretq_s16_u32(reg0); \
1869 int16x8_t r1 = vreinterpretq_s16_u32(reg1); \
1870 int32x4_t tmp0 = vmull_s16( vget_low_s16(r0), vget_low_s16(r1) ); \
1871 int32x4_t tmp1 = vmull_s16( vget_high_s16(r0), vget_high_s16(r1) ); \
1872 int32x2_t out0 = vpadd_s32( vget_low_s32(tmp0), vget_high_s32(tmp0) ); \
1873 int32x2_t out1 = vpadd_s32( vget_low_s32(tmp1), vget_high_s32(tmp1) ); \
1874 (out) = vreinterpretq_u32_s32( vcombine_s32(out0, out1) ); \
1875 }
1876
1877 #endif
1878
1879 #define stbir__simdi_and( out, reg0, reg1 ) (out) = vandq_u32( reg0, reg1 )
1880 #define stbir__simdi_or( out, reg0, reg1 ) (out) = vorrq_u32( reg0, reg1 )
1881
1882 #define stbir__simdf_pack_to_8bytes(out,aa,bb) \
1883 { \
1884 float32x4_t af = vmaxq_f32( vminq_f32(aa,STBIR__CONSTF(STBIR_max_uint8_as_float) ), vdupq_n_f32(0) ); \
1885 float32x4_t bf = vmaxq_f32( vminq_f32(bb,STBIR__CONSTF(STBIR_max_uint8_as_float) ), vdupq_n_f32(0) ); \
1886 int16x4_t ai = vqmovn_s32( vcvtq_s32_f32( af ) ); \
1887 int16x4_t bi = vqmovn_s32( vcvtq_s32_f32( bf ) ); \
1888 uint8x8_t out8 = vqmovun_s16( vcombine_s16(ai, bi) ); \
1889 out = vreinterpretq_u32_u8( vcombine_u8(out8, out8) ); \
1890 }
1891
1892 #define stbir__simdf_pack_to_8words(out,aa,bb) \
1893 { \
1894 float32x4_t af = vmaxq_f32( vminq_f32(aa,STBIR__CONSTF(STBIR_max_uint16_as_float) ), vdupq_n_f32(0) ); \
1895 float32x4_t bf = vmaxq_f32( vminq_f32(bb,STBIR__CONSTF(STBIR_max_uint16_as_float) ), vdupq_n_f32(0) ); \
1896 int32x4_t ai = vcvtq_s32_f32( af ); \
1897 int32x4_t bi = vcvtq_s32_f32( bf ); \
1898 out = vreinterpretq_u32_u16( vcombine_u16(vqmovun_s32(ai), vqmovun_s32(bi)) ); \
1899 }
1900
1901 #define stbir__interleave_pack_and_store_16_u8( ptr, r0, r1, r2, r3 ) \
1902 { \
1903 int16x4x2_t tmp0 = vzip_s16( vqmovn_s32(vreinterpretq_s32_u32(r0)), vqmovn_s32(vreinterpretq_s32_u32(r2)) ); \
1904 int16x4x2_t tmp1 = vzip_s16( vqmovn_s32(vreinterpretq_s32_u32(r1)), vqmovn_s32(vreinterpretq_s32_u32(r3)) ); \
1905 uint8x8x2_t out = \
1906 { { \
1907 vqmovun_s16( vcombine_s16(tmp0.val[0], tmp0.val[1]) ), \
1908 vqmovun_s16( vcombine_s16(tmp1.val[0], tmp1.val[1]) ), \
1909 } }; \
1910 vst2_u8(ptr, out); \
1911 }
1912
1913 #define stbir__simdf_load4_transposed( o0, o1, o2, o3, ptr ) \
1914 { \
1915 float32x4x4_t tmp = vld4q_f32(ptr); \
1916 o0 = tmp.val[0]; \
1917 o1 = tmp.val[1]; \
1918 o2 = tmp.val[2]; \
1919 o3 = tmp.val[3]; \
1920 }
1921
1922 #define stbir__simdi_32shr( out, reg, imm ) out = vshrq_n_u32( reg, imm )
1923
1924 #if defined( _MSC_VER ) && !defined(__clang__)
1925 #define STBIR__SIMDF_CONST(var, x) __declspec(align(8)) float var[] = { x, x, x, x }
1926 #define STBIR__SIMDI_CONST(var, x) __declspec(align(8)) uint32_t var[] = { x, x, x, x }
1927 #define STBIR__CONSTF(var) (*(const float32x4_t*)var)
1928 #define STBIR__CONSTI(var) (*(const uint32x4_t*)var)
1929 #else
1930 #define STBIR__SIMDF_CONST(var, x) stbir__simdf var = { x, x, x, x }
1931 #define STBIR__SIMDI_CONST(var, x) stbir__simdi var = { x, x, x, x }
1932 #define STBIR__CONSTF(var) (var)
1933 #define STBIR__CONSTI(var) (var)
1934 #endif
1935
1936 #ifdef STBIR_FLOORF
1937 #undef STBIR_FLOORF
1938 #endif
1939 #define STBIR_FLOORF stbir_simd_floorf
1940 static stbir__inline float stbir_simd_floorf(float x)
1941 {
1942 #if defined( _M_ARM64 ) || defined( __aarch64__ ) || defined( __arm64__ )
1943 return vget_lane_f32( vrndm_f32( vdup_n_f32(x) ), 0);
1944 #else
1945 float32x2_t f = vdup_n_f32(x);
1946 float32x2_t t = vcvt_f32_s32(vcvt_s32_f32(f));
1947 uint32x2_t a = vclt_f32(f, t);
1948 uint32x2_t b = vreinterpret_u32_f32(vdup_n_f32(-1.0f));
1949 float32x2_t r = vadd_f32(t, vreinterpret_f32_u32(vand_u32(a, b)));
1950 return vget_lane_f32(r, 0);
1951 #endif
1952 }
1953
1954 #ifdef STBIR_CEILF
1955 #undef STBIR_CEILF
1956 #endif
1957 #define STBIR_CEILF stbir_simd_ceilf
1958 static stbir__inline float stbir_simd_ceilf(float x)
1959 {
1960 #if defined( _M_ARM64 ) || defined( __aarch64__ ) || defined( __arm64__ )
1961 return vget_lane_f32( vrndp_f32( vdup_n_f32(x) ), 0);
1962 #else
1963 float32x2_t f = vdup_n_f32(x);
1964 float32x2_t t = vcvt_f32_s32(vcvt_s32_f32(f));
1965 uint32x2_t a = vclt_f32(t, f);
1966 uint32x2_t b = vreinterpret_u32_f32(vdup_n_f32(1.0f));
1967 float32x2_t r = vadd_f32(t, vreinterpret_f32_u32(vand_u32(a, b)));
1968 return vget_lane_f32(r, 0);
1969 #endif
1970 }
1971
1972 #define STBIR_SIMD
1973
1974#elif defined(STBIR_WASM)
1975
1976 #include <wasm_simd128.h>
1977
1978 #define stbir__simdf v128_t
1979 #define stbir__simdi v128_t
1980
1981 #define stbir_simdi_castf( reg ) (reg)
1982 #define stbir_simdf_casti( reg ) (reg)
1983
1984 #define stbir__simdf_load( reg, ptr ) (reg) = wasm_v128_load( (void const*)(ptr) )
1985 #define stbir__simdi_load( reg, ptr ) (reg) = wasm_v128_load( (void const*)(ptr) )
1986 #define stbir__simdf_load1( out, ptr ) (out) = wasm_v128_load32_splat( (void const*)(ptr) ) // top values can be random (not denormal or nan for perf)
1987 #define stbir__simdi_load1( out, ptr ) (out) = wasm_v128_load32_splat( (void const*)(ptr) )
1988 #define stbir__simdf_load1z( out, ptr ) (out) = wasm_v128_load32_zero( (void const*)(ptr) ) // top values must be zero
1989 #define stbir__simdf_frep4( fvar ) wasm_f32x4_splat( fvar )
1990 #define stbir__simdf_load1frep4( out, fvar ) (out) = wasm_f32x4_splat( fvar )
1991 #define stbir__simdf_load2( out, ptr ) (out) = wasm_v128_load64_splat( (void const*)(ptr) ) // top values can be random (not denormal or nan for perf)
1992 #define stbir__simdf_load2z( out, ptr ) (out) = wasm_v128_load64_zero( (void const*)(ptr) ) // top values must be zero
1993 #define stbir__simdf_load2hmerge( out, reg, ptr ) (out) = wasm_v128_load64_lane( (void const*)(ptr), reg, 1 )
1994
1995 #define stbir__simdf_zeroP() wasm_f32x4_const_splat(0)
1996 #define stbir__simdf_zero( reg ) (reg) = wasm_f32x4_const_splat(0)
1997
1998 #define stbir__simdf_store( ptr, reg ) wasm_v128_store( (void*)(ptr), reg )
1999 #define stbir__simdf_store1( ptr, reg ) wasm_v128_store32_lane( (void*)(ptr), reg, 0 )
2000 #define stbir__simdf_store2( ptr, reg ) wasm_v128_store64_lane( (void*)(ptr), reg, 0 )
2001 #define stbir__simdf_store2h( ptr, reg ) wasm_v128_store64_lane( (void*)(ptr), reg, 1 )
2002
2003 #define stbir__simdi_store( ptr, reg ) wasm_v128_store( (void*)(ptr), reg )
2004 #define stbir__simdi_store1( ptr, reg ) wasm_v128_store32_lane( (void*)(ptr), reg, 0 )
2005 #define stbir__simdi_store2( ptr, reg ) wasm_v128_store64_lane( (void*)(ptr), reg, 0 )
2006
2007 #define stbir__prefetch( ptr )
2008
2009 #define stbir__simdi_expand_u8_to_u32(out0,out1,out2,out3,ireg) \
2010 { \
2011 v128_t l = wasm_u16x8_extend_low_u8x16 ( ireg ); \
2012 v128_t h = wasm_u16x8_extend_high_u8x16( ireg ); \
2013 out0 = wasm_u32x4_extend_low_u16x8 ( l ); \
2014 out1 = wasm_u32x4_extend_high_u16x8( l ); \
2015 out2 = wasm_u32x4_extend_low_u16x8 ( h ); \
2016 out3 = wasm_u32x4_extend_high_u16x8( h ); \
2017 }
2018
2019 #define stbir__simdi_expand_u8_to_1u32(out,ireg) \
2020 { \
2021 v128_t tmp = wasm_u16x8_extend_low_u8x16(ireg); \
2022 out = wasm_u32x4_extend_low_u16x8(tmp); \
2023 }
2024
2025 #define stbir__simdi_expand_u16_to_u32(out0,out1,ireg) \
2026 { \
2027 out0 = wasm_u32x4_extend_low_u16x8 ( ireg ); \
2028 out1 = wasm_u32x4_extend_high_u16x8( ireg ); \
2029 }
2030
2031 #define stbir__simdf_convert_float_to_i32( i, f ) (i) = wasm_i32x4_trunc_sat_f32x4(f)
2032 #define stbir__simdf_convert_float_to_int( f ) wasm_i32x4_extract_lane(wasm_i32x4_trunc_sat_f32x4(f), 0)
2033 #define stbir__simdi_to_int( i ) wasm_i32x4_extract_lane(i, 0)
2034 #define stbir__simdf_convert_float_to_uint8( f ) ((unsigned char)wasm_i32x4_extract_lane(wasm_i32x4_trunc_sat_f32x4(wasm_f32x4_max(wasm_f32x4_min(f,STBIR_max_uint8_as_float),wasm_f32x4_const_splat(0))), 0))
2035 #define stbir__simdf_convert_float_to_short( f ) ((unsigned short)wasm_i32x4_extract_lane(wasm_i32x4_trunc_sat_f32x4(wasm_f32x4_max(wasm_f32x4_min(f,STBIR_max_uint16_as_float),wasm_f32x4_const_splat(0))), 0))
2036 #define stbir__simdi_convert_i32_to_float(out, ireg) (out) = wasm_f32x4_convert_i32x4(ireg)
2037 #define stbir__simdf_add( out, reg0, reg1 ) (out) = wasm_f32x4_add( reg0, reg1 )
2038 #define stbir__simdf_mult( out, reg0, reg1 ) (out) = wasm_f32x4_mul( reg0, reg1 )
2039 #define stbir__simdf_mult_mem( out, reg, ptr ) (out) = wasm_f32x4_mul( reg, wasm_v128_load( (void const*)(ptr) ) )
2040 #define stbir__simdf_mult1_mem( out, reg, ptr ) (out) = wasm_f32x4_mul( reg, wasm_v128_load32_splat( (void const*)(ptr) ) )
2041 #define stbir__simdf_add_mem( out, reg, ptr ) (out) = wasm_f32x4_add( reg, wasm_v128_load( (void const*)(ptr) ) )
2042 #define stbir__simdf_add1_mem( out, reg, ptr ) (out) = wasm_f32x4_add( reg, wasm_v128_load32_splat( (void const*)(ptr) ) )
2043
2044 #define stbir__simdf_madd( out, add, mul1, mul2 ) (out) = wasm_f32x4_add( add, wasm_f32x4_mul( mul1, mul2 ) )
2045 #define stbir__simdf_madd1( out, add, mul1, mul2 ) (out) = wasm_f32x4_add( add, wasm_f32x4_mul( mul1, mul2 ) )
2046 #define stbir__simdf_madd_mem( out, add, mul, ptr ) (out) = wasm_f32x4_add( add, wasm_f32x4_mul( mul, wasm_v128_load( (void const*)(ptr) ) ) )
2047 #define stbir__simdf_madd1_mem( out, add, mul, ptr ) (out) = wasm_f32x4_add( add, wasm_f32x4_mul( mul, wasm_v128_load32_splat( (void const*)(ptr) ) ) )
2048
2049 #define stbir__simdf_add1( out, reg0, reg1 ) (out) = wasm_f32x4_add( reg0, reg1 )
2050 #define stbir__simdf_mult1( out, reg0, reg1 ) (out) = wasm_f32x4_mul( reg0, reg1 )
2051
2052 #define stbir__simdf_and( out, reg0, reg1 ) (out) = wasm_v128_and( reg0, reg1 )
2053 #define stbir__simdf_or( out, reg0, reg1 ) (out) = wasm_v128_or( reg0, reg1 )
2054
2055 #define stbir__simdf_min( out, reg0, reg1 ) (out) = wasm_f32x4_min( reg0, reg1 )
2056 #define stbir__simdf_max( out, reg0, reg1 ) (out) = wasm_f32x4_max( reg0, reg1 )
2057 #define stbir__simdf_min1( out, reg0, reg1 ) (out) = wasm_f32x4_min( reg0, reg1 )
2058 #define stbir__simdf_max1( out, reg0, reg1 ) (out) = wasm_f32x4_max( reg0, reg1 )
2059
2060 #define stbir__simdf_0123ABCDto3ABx( out, reg0, reg1 ) (out) = wasm_i32x4_shuffle( reg0, reg1, 3, 4, 5, -1 )
2061 #define stbir__simdf_0123ABCDto23Ax( out, reg0, reg1 ) (out) = wasm_i32x4_shuffle( reg0, reg1, 2, 3, 4, -1 )
2062
2063 #define stbir__simdf_aaa1(out,alp,ones) (out) = wasm_i32x4_shuffle(alp, ones, 3, 3, 3, 4)
2064 #define stbir__simdf_1aaa(out,alp,ones) (out) = wasm_i32x4_shuffle(alp, ones, 4, 0, 0, 0)
2065 #define stbir__simdf_a1a1(out,alp,ones) (out) = wasm_i32x4_shuffle(alp, ones, 1, 4, 3, 4)
2066 #define stbir__simdf_1a1a(out,alp,ones) (out) = wasm_i32x4_shuffle(alp, ones, 4, 0, 4, 2)
2067
2068 #define stbir__simdf_swiz( reg, one, two, three, four ) wasm_i32x4_shuffle(reg, reg, one, two, three, four)
2069
2070 #define stbir__simdi_and( out, reg0, reg1 ) (out) = wasm_v128_and( reg0, reg1 )
2071 #define stbir__simdi_or( out, reg0, reg1 ) (out) = wasm_v128_or( reg0, reg1 )
2072 #define stbir__simdi_16madd( out, reg0, reg1 ) (out) = wasm_i32x4_dot_i16x8( reg0, reg1 )
2073
2074 #define stbir__simdf_pack_to_8bytes(out,aa,bb) \
2075 { \
2076 v128_t af = wasm_f32x4_max( wasm_f32x4_min(aa, STBIR_max_uint8_as_float), wasm_f32x4_const_splat(0) ); \
2077 v128_t bf = wasm_f32x4_max( wasm_f32x4_min(bb, STBIR_max_uint8_as_float), wasm_f32x4_const_splat(0) ); \
2078 v128_t ai = wasm_i32x4_trunc_sat_f32x4( af ); \
2079 v128_t bi = wasm_i32x4_trunc_sat_f32x4( bf ); \
2080 v128_t out16 = wasm_i16x8_narrow_i32x4( ai, bi ); \
2081 out = wasm_u8x16_narrow_i16x8( out16, out16 ); \
2082 }
2083
2084 #define stbir__simdf_pack_to_8words(out,aa,bb) \
2085 { \
2086 v128_t af = wasm_f32x4_max( wasm_f32x4_min(aa, STBIR_max_uint16_as_float), wasm_f32x4_const_splat(0)); \
2087 v128_t bf = wasm_f32x4_max( wasm_f32x4_min(bb, STBIR_max_uint16_as_float), wasm_f32x4_const_splat(0)); \
2088 v128_t ai = wasm_i32x4_trunc_sat_f32x4( af ); \
2089 v128_t bi = wasm_i32x4_trunc_sat_f32x4( bf ); \
2090 out = wasm_u16x8_narrow_i32x4( ai, bi ); \
2091 }
2092
2093 #define stbir__interleave_pack_and_store_16_u8( ptr, r0, r1, r2, r3 ) \
2094 { \
2095 v128_t tmp0 = wasm_i16x8_narrow_i32x4(r0, r1); \
2096 v128_t tmp1 = wasm_i16x8_narrow_i32x4(r2, r3); \
2097 v128_t tmp = wasm_u8x16_narrow_i16x8(tmp0, tmp1); \
2098 tmp = wasm_i8x16_shuffle(tmp, tmp, 0, 4, 8, 12, 1, 5, 9, 13, 2, 6, 10, 14, 3, 7, 11, 15); \
2099 wasm_v128_store( (void*)(ptr), tmp); \
2100 }
2101
2102 #define stbir__simdf_load4_transposed( o0, o1, o2, o3, ptr ) \
2103 { \
2104 v128_t t0 = wasm_v128_load( ptr ); \
2105 v128_t t1 = wasm_v128_load( ptr+4 ); \
2106 v128_t t2 = wasm_v128_load( ptr+8 ); \
2107 v128_t t3 = wasm_v128_load( ptr+12 ); \
2108 v128_t s0 = wasm_i32x4_shuffle(t0, t1, 0, 4, 2, 6); \
2109 v128_t s1 = wasm_i32x4_shuffle(t0, t1, 1, 5, 3, 7); \
2110 v128_t s2 = wasm_i32x4_shuffle(t2, t3, 0, 4, 2, 6); \
2111 v128_t s3 = wasm_i32x4_shuffle(t2, t3, 1, 5, 3, 7); \
2112 o0 = wasm_i32x4_shuffle(s0, s2, 0, 1, 4, 5); \
2113 o1 = wasm_i32x4_shuffle(s1, s3, 0, 1, 4, 5); \
2114 o2 = wasm_i32x4_shuffle(s0, s2, 2, 3, 6, 7); \
2115 o3 = wasm_i32x4_shuffle(s1, s3, 2, 3, 6, 7); \
2116 }
2117
2118 #define stbir__simdi_32shr( out, reg, imm ) out = wasm_u32x4_shr( reg, imm )
2119
2120 typedef float stbir__f32x4 __attribute__((__vector_size__(16), __aligned__(16)));
2121 #define STBIR__SIMDF_CONST(var, x) stbir__simdf var = (v128_t)(stbir__f32x4){ x, x, x, x }
2122 #define STBIR__SIMDI_CONST(var, x) stbir__simdi var = { x, x, x, x }
2123 #define STBIR__CONSTF(var) (var)
2124 #define STBIR__CONSTI(var) (var)
2125
2126 #ifdef STBIR_FLOORF
2127 #undef STBIR_FLOORF
2128 #endif
2129 #define STBIR_FLOORF stbir_simd_floorf
2130 static stbir__inline float stbir_simd_floorf(float x)
2131 {
2132 return wasm_f32x4_extract_lane( wasm_f32x4_floor( wasm_f32x4_splat(x) ), 0);
2133 }
2134
2135 #ifdef STBIR_CEILF
2136 #undef STBIR_CEILF
2137 #endif
2138 #define STBIR_CEILF stbir_simd_ceilf
2139 static stbir__inline float stbir_simd_ceilf(float x)
2140 {
2141 return wasm_f32x4_extract_lane( wasm_f32x4_ceil( wasm_f32x4_splat(x) ), 0);
2142 }
2143
2144 #define STBIR_SIMD
2145
2146#endif // SSE2/NEON/WASM
2147
2148#endif // NO SIMD
2149
2150#ifdef STBIR_SIMD8
2151 #define stbir__simdfX stbir__simdf8
2152 #define stbir__simdiX stbir__simdi8
2153 #define stbir__simdfX_load stbir__simdf8_load
2154 #define stbir__simdiX_load stbir__simdi8_load
2155 #define stbir__simdfX_mult stbir__simdf8_mult
2156 #define stbir__simdfX_add_mem stbir__simdf8_add_mem
2157 #define stbir__simdfX_madd_mem stbir__simdf8_madd_mem
2158 #define stbir__simdfX_store stbir__simdf8_store
2159 #define stbir__simdiX_store stbir__simdi8_store
2160 #define stbir__simdf_frepX stbir__simdf8_frep8
2161 #define stbir__simdfX_madd stbir__simdf8_madd
2162 #define stbir__simdfX_min stbir__simdf8_min
2163 #define stbir__simdfX_max stbir__simdf8_max
2164 #define stbir__simdfX_aaa1 stbir__simdf8_aaa1
2165 #define stbir__simdfX_1aaa stbir__simdf8_1aaa
2166 #define stbir__simdfX_a1a1 stbir__simdf8_a1a1
2167 #define stbir__simdfX_1a1a stbir__simdf8_1a1a
2168 #define stbir__simdfX_convert_float_to_i32 stbir__simdf8_convert_float_to_i32
2169 #define stbir__simdfX_pack_to_words stbir__simdf8_pack_to_16words
2170 #define stbir__simdfX_zero stbir__simdf8_zero
2171 #define STBIR_onesX STBIR_ones8
2172 #define STBIR_max_uint8_as_floatX STBIR_max_uint8_as_float8
2173 #define STBIR_max_uint16_as_floatX STBIR_max_uint16_as_float8
2174 #define STBIR_simd_point5X STBIR_simd_point58
2175 #define stbir__simdfX_float_count 8
2176 #define stbir__simdfX_0123to1230 stbir__simdf8_0123to12301230
2177 #define stbir__simdfX_0123to2103 stbir__simdf8_0123to21032103
2178 static const stbir__simdf8 STBIR_max_uint16_as_float_inverted8 = { stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted,stbir__max_uint16_as_float_inverted };
2179 static const stbir__simdf8 STBIR_max_uint8_as_float_inverted8 = { stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted,stbir__max_uint8_as_float_inverted };
2180 static const stbir__simdf8 STBIR_ones8 = { 1.0,1.0,1.0,1.0,1.0,1.0,1.0,1.0 };
2181 static const stbir__simdf8 STBIR_simd_point58 = { 0.5,0.5,0.5,0.5,0.5,0.5,0.5,0.5 };
2182 static const stbir__simdf8 STBIR_max_uint8_as_float8 = { stbir__max_uint8_as_float,stbir__max_uint8_as_float,stbir__max_uint8_as_float,stbir__max_uint8_as_float, stbir__max_uint8_as_float,stbir__max_uint8_as_float,stbir__max_uint8_as_float,stbir__max_uint8_as_float };
2183 static const stbir__simdf8 STBIR_max_uint16_as_float8 = { stbir__max_uint16_as_float,stbir__max_uint16_as_float,stbir__max_uint16_as_float,stbir__max_uint16_as_float, stbir__max_uint16_as_float,stbir__max_uint16_as_float,stbir__max_uint16_as_float,stbir__max_uint16_as_float };
2184#else
2185 #define stbir__simdfX stbir__simdf
2186 #define stbir__simdiX stbir__simdi
2187 #define stbir__simdfX_load stbir__simdf_load
2188 #define stbir__simdiX_load stbir__simdi_load
2189 #define stbir__simdfX_mult stbir__simdf_mult
2190 #define stbir__simdfX_add_mem stbir__simdf_add_mem
2191 #define stbir__simdfX_madd_mem stbir__simdf_madd_mem
2192 #define stbir__simdfX_store stbir__simdf_store
2193 #define stbir__simdiX_store stbir__simdi_store
2194 #define stbir__simdf_frepX stbir__simdf_frep4
2195 #define stbir__simdfX_madd stbir__simdf_madd
2196 #define stbir__simdfX_min stbir__simdf_min
2197 #define stbir__simdfX_max stbir__simdf_max
2198 #define stbir__simdfX_aaa1 stbir__simdf_aaa1
2199 #define stbir__simdfX_1aaa stbir__simdf_1aaa
2200 #define stbir__simdfX_a1a1 stbir__simdf_a1a1
2201 #define stbir__simdfX_1a1a stbir__simdf_1a1a
2202 #define stbir__simdfX_convert_float_to_i32 stbir__simdf_convert_float_to_i32
2203 #define stbir__simdfX_pack_to_words stbir__simdf_pack_to_8words
2204 #define stbir__simdfX_zero stbir__simdf_zero
2205 #define STBIR_onesX STBIR__CONSTF(STBIR_ones)
2206 #define STBIR_simd_point5X STBIR__CONSTF(STBIR_simd_point5)
2207 #define STBIR_max_uint8_as_floatX STBIR__CONSTF(STBIR_max_uint8_as_float)
2208 #define STBIR_max_uint16_as_floatX STBIR__CONSTF(STBIR_max_uint16_as_float)
2209 #define stbir__simdfX_float_count 4
2210 #define stbir__if_simdf8_cast_to_simdf4( val ) ( val )
2211 #define stbir__simdfX_0123to1230 stbir__simdf_0123to1230
2212 #define stbir__simdfX_0123to2103 stbir__simdf_0123to2103
2213#endif
2214
2215
2216#if defined(STBIR_NEON) && !defined(_M_ARM) && !defined(__arm__)
2217
2218 #if defined( _MSC_VER ) && !defined(__clang__)
2219 typedef __int16 stbir__FP16;
2220 #else
2221 typedef float16_t stbir__FP16;
2222 #endif
2223
2224#else // no NEON, or 32-bit ARM for MSVC
2225
2226 typedef union stbir__FP16
2227 {
2228 unsigned short u;
2229 } stbir__FP16;
2230
2231#endif
2232
2233#if (!defined(STBIR_NEON) && !defined(STBIR_FP16C)) || (defined(STBIR_NEON) && defined(_M_ARM)) || (defined(STBIR_NEON) && defined(__arm__))
2234
2235 // Fabian's half float routines, see: https://gist.github.com/rygorous/2156668
2236
2237 static stbir__inline float stbir__half_to_float( stbir__FP16 h )
2238 {
2239 static const stbir__FP32 magic = { (254 - 15) << 23 };
2240 static const stbir__FP32 was_infnan = { (127 + 16) << 23 };
2241 stbir__FP32 o;
2242
2243 o.u = (h.u & 0x7fff) << 13; // exponent/mantissa bits
2244 o.f *= magic.f; // exponent adjust
2245 if (o.f >= was_infnan.f) // make sure Inf/NaN survive
2246 o.u |= 255 << 23;
2247 o.u |= (h.u & 0x8000) << 16; // sign bit
2248 return o.f;
2249 }
2250
2251 static stbir__inline stbir__FP16 stbir__float_to_half(float val)
2252 {
2253 stbir__FP32 f32infty = { 255 << 23 };
2254 stbir__FP32 f16max = { (127 + 16) << 23 };
2255 stbir__FP32 denorm_magic = { ((127 - 15) + (23 - 10) + 1) << 23 };
2256 unsigned int sign_mask = 0x80000000u;
2257 stbir__FP16 o = { 0 };
2258 stbir__FP32 f;
2259 unsigned int sign;
2260
2261 f.f = val;
2262 sign = f.u & sign_mask;
2263 f.u ^= sign;
2264
2265 if (f.u >= f16max.u) // result is Inf or NaN (all exponent bits set)
2266 o.u = (f.u > f32infty.u) ? 0x7e00 : 0x7c00; // NaN->qNaN and Inf->Inf
2267 else // (De)normalized number or zero
2268 {
2269 if (f.u < (113 << 23)) // resulting FP16 is subnormal or zero
2270 {
2271 // use a magic value to align our 10 mantissa bits at the bottom of
2272 // the float. as long as FP addition is round-to-nearest-even this
2273 // just works.
2274 f.f += denorm_magic.f;
2275 // and one integer subtract of the bias later, we have our final float!
2276 o.u = (unsigned short) ( f.u - denorm_magic.u );
2277 }
2278 else
2279 {
2280 unsigned int mant_odd = (f.u >> 13) & 1; // resulting mantissa is odd
2281 // update exponent, rounding bias part 1
2282 f.u = f.u + ((15u - 127) << 23) + 0xfff;
2283 // rounding bias part 2
2284 f.u += mant_odd;
2285 // take the bits!
2286 o.u = (unsigned short) ( f.u >> 13 );
2287 }
2288 }
2289
2290 o.u |= sign >> 16;
2291 return o;
2292 }
2293
2294#endif
2295
2296
2297#if defined(STBIR_FP16C)
2298
2299 #include <immintrin.h>
2300
2301 static stbir__inline void stbir__half_to_float_SIMD(float * output, stbir__FP16 const * input)
2302 {
2303 _mm256_storeu_ps( (float*)output, _mm256_cvtph_ps( _mm_loadu_si128( (__m128i const* )input ) ) );
2304 }
2305
2306 static stbir__inline void stbir__float_to_half_SIMD(stbir__FP16 * output, float const * input)
2307 {
2308 _mm_storeu_si128( (__m128i*)output, _mm256_cvtps_ph( _mm256_loadu_ps( input ), 0 ) );
2309 }
2310
2311 static stbir__inline float stbir__half_to_float( stbir__FP16 h )
2312 {
2313 return _mm_cvtss_f32( _mm_cvtph_ps( _mm_cvtsi32_si128( (int)h.u ) ) );
2314 }
2315
2316 static stbir__inline stbir__FP16 stbir__float_to_half( float f )
2317 {
2318 stbir__FP16 h;
2319 h.u = (unsigned short) _mm_cvtsi128_si32( _mm_cvtps_ph( _mm_set_ss( f ), 0 ) );
2320 return h;
2321 }
2322
2323#elif defined(STBIR_SSE2)
2324
2325 // Fabian's half float routines, see: https://gist.github.com/rygorous/2156668
2326 stbir__inline static void stbir__half_to_float_SIMD(float * output, void const * input)
2327 {
2328 static const STBIR__SIMDI_CONST(mask_nosign, 0x7fff);
2329 static const STBIR__SIMDI_CONST(smallest_normal, 0x0400);
2330 static const STBIR__SIMDI_CONST(infinity, 0x7c00);
2331 static const STBIR__SIMDI_CONST(expadjust_normal, (127 - 15) << 23);
2332 static const STBIR__SIMDI_CONST(magic_denorm, 113 << 23);
2333
2334 __m128i i = _mm_loadu_si128 ( (__m128i const*)(input) );
2335 __m128i h = _mm_unpacklo_epi16 ( i, _mm_setzero_si128() );
2336 __m128i mnosign = STBIR__CONSTI(mask_nosign);
2337 __m128i eadjust = STBIR__CONSTI(expadjust_normal);
2338 __m128i smallest = STBIR__CONSTI(smallest_normal);
2339 __m128i infty = STBIR__CONSTI(infinity);
2340 __m128i expmant = _mm_and_si128(mnosign, h);
2341 __m128i justsign = _mm_xor_si128(h, expmant);
2342 __m128i b_notinfnan = _mm_cmpgt_epi32(infty, expmant);
2343 __m128i b_isdenorm = _mm_cmpgt_epi32(smallest, expmant);
2344 __m128i shifted = _mm_slli_epi32(expmant, 13);
2345 __m128i adj_infnan = _mm_andnot_si128(b_notinfnan, eadjust);
2346 __m128i adjusted = _mm_add_epi32(eadjust, shifted);
2347 __m128i den1 = _mm_add_epi32(shifted, STBIR__CONSTI(magic_denorm));
2348 __m128i adjusted2 = _mm_add_epi32(adjusted, adj_infnan);
2349 __m128 den2 = _mm_sub_ps(_mm_castsi128_ps(den1), *(const __m128 *)&magic_denorm);
2350 __m128 adjusted3 = _mm_and_ps(den2, _mm_castsi128_ps(b_isdenorm));
2351 __m128 adjusted4 = _mm_andnot_ps(_mm_castsi128_ps(b_isdenorm), _mm_castsi128_ps(adjusted2));
2352 __m128 adjusted5 = _mm_or_ps(adjusted3, adjusted4);
2353 __m128i sign = _mm_slli_epi32(justsign, 16);
2354 __m128 final = _mm_or_ps(adjusted5, _mm_castsi128_ps(sign));
2355 stbir__simdf_store( output + 0, final );
2356
2357 h = _mm_unpackhi_epi16 ( i, _mm_setzero_si128() );
2358 expmant = _mm_and_si128(mnosign, h);
2359 justsign = _mm_xor_si128(h, expmant);
2360 b_notinfnan = _mm_cmpgt_epi32(infty, expmant);
2361 b_isdenorm = _mm_cmpgt_epi32(smallest, expmant);
2362 shifted = _mm_slli_epi32(expmant, 13);
2363 adj_infnan = _mm_andnot_si128(b_notinfnan, eadjust);
2364 adjusted = _mm_add_epi32(eadjust, shifted);
2365 den1 = _mm_add_epi32(shifted, STBIR__CONSTI(magic_denorm));
2366 adjusted2 = _mm_add_epi32(adjusted, adj_infnan);
2367 den2 = _mm_sub_ps(_mm_castsi128_ps(den1), *(const __m128 *)&magic_denorm);
2368 adjusted3 = _mm_and_ps(den2, _mm_castsi128_ps(b_isdenorm));
2369 adjusted4 = _mm_andnot_ps(_mm_castsi128_ps(b_isdenorm), _mm_castsi128_ps(adjusted2));
2370 adjusted5 = _mm_or_ps(adjusted3, adjusted4);
2371 sign = _mm_slli_epi32(justsign, 16);
2372 final = _mm_or_ps(adjusted5, _mm_castsi128_ps(sign));
2373 stbir__simdf_store( output + 4, final );
2374
2375 // ~38 SSE2 ops for 8 values
2376 }
2377
2378 // Fabian's round-to-nearest-even float to half
2379 // ~48 SSE2 ops for 8 output
2380 stbir__inline static void stbir__float_to_half_SIMD(void * output, float const * input)
2381 {
2382 static const STBIR__SIMDI_CONST(mask_sign, 0x80000000u);
2383 static const STBIR__SIMDI_CONST(c_f16max, (127 + 16) << 23); // all FP32 values >=this round to +inf
2384 static const STBIR__SIMDI_CONST(c_nanbit, 0x200);
2385 static const STBIR__SIMDI_CONST(c_infty_as_fp16, 0x7c00);
2386 static const STBIR__SIMDI_CONST(c_min_normal, (127 - 14) << 23); // smallest FP32 that yields a normalized FP16
2387 static const STBIR__SIMDI_CONST(c_subnorm_magic, ((127 - 15) + (23 - 10) + 1) << 23);
2388 static const STBIR__SIMDI_CONST(c_normal_bias, 0xfff - ((127 - 15) << 23)); // adjust exponent and add mantissa rounding
2389
2390 __m128 f = _mm_loadu_ps(input);
2391 __m128 msign = _mm_castsi128_ps(STBIR__CONSTI(mask_sign));
2392 __m128 justsign = _mm_and_ps(msign, f);
2393 __m128 absf = _mm_xor_ps(f, justsign);
2394 __m128i absf_int = _mm_castps_si128(absf); // the cast is "free" (extra bypass latency, but no thruput hit)
2395 __m128i f16max = STBIR__CONSTI(c_f16max);
2396 __m128 b_isnan = _mm_cmpunord_ps(absf, absf); // is this a NaN?
2397 __m128i b_isregular = _mm_cmpgt_epi32(f16max, absf_int); // (sub)normalized or special?
2398 __m128i nanbit = _mm_and_si128(_mm_castps_si128(b_isnan), STBIR__CONSTI(c_nanbit));
2399 __m128i inf_or_nan = _mm_or_si128(nanbit, STBIR__CONSTI(c_infty_as_fp16)); // output for specials
2400
2401 __m128i min_normal = STBIR__CONSTI(c_min_normal);
2402 __m128i b_issub = _mm_cmpgt_epi32(min_normal, absf_int);
2403
2404 // "result is subnormal" path
2405 __m128 subnorm1 = _mm_add_ps(absf, _mm_castsi128_ps(STBIR__CONSTI(c_subnorm_magic))); // magic value to round output mantissa
2406 __m128i subnorm2 = _mm_sub_epi32(_mm_castps_si128(subnorm1), STBIR__CONSTI(c_subnorm_magic)); // subtract out bias
2407
2408 // "result is normal" path
2409 __m128i mantoddbit = _mm_slli_epi32(absf_int, 31 - 13); // shift bit 13 (mantissa LSB) to sign
2410 __m128i mantodd = _mm_srai_epi32(mantoddbit, 31); // -1 if FP16 mantissa odd, else 0
2411
2412 __m128i round1 = _mm_add_epi32(absf_int, STBIR__CONSTI(c_normal_bias));
2413 __m128i round2 = _mm_sub_epi32(round1, mantodd); // if mantissa LSB odd, bias towards rounding up (RTNE)
2414 __m128i normal = _mm_srli_epi32(round2, 13); // rounded result
2415
2416 // combine the two non-specials
2417 __m128i nonspecial = _mm_or_si128(_mm_and_si128(subnorm2, b_issub), _mm_andnot_si128(b_issub, normal));
2418
2419 // merge in specials as well
2420 __m128i joined = _mm_or_si128(_mm_and_si128(nonspecial, b_isregular), _mm_andnot_si128(b_isregular, inf_or_nan));
2421
2422 __m128i sign_shift = _mm_srai_epi32(_mm_castps_si128(justsign), 16);
2423 __m128i final2, final= _mm_or_si128(joined, sign_shift);
2424
2425 f = _mm_loadu_ps(input+4);
2426 justsign = _mm_and_ps(msign, f);
2427 absf = _mm_xor_ps(f, justsign);
2428 absf_int = _mm_castps_si128(absf); // the cast is "free" (extra bypass latency, but no thruput hit)
2429 b_isnan = _mm_cmpunord_ps(absf, absf); // is this a NaN?
2430 b_isregular = _mm_cmpgt_epi32(f16max, absf_int); // (sub)normalized or special?
2431 nanbit = _mm_and_si128(_mm_castps_si128(b_isnan), c_nanbit);
2432 inf_or_nan = _mm_or_si128(nanbit, STBIR__CONSTI(c_infty_as_fp16)); // output for specials
2433
2434 b_issub = _mm_cmpgt_epi32(min_normal, absf_int);
2435
2436 // "result is subnormal" path
2437 subnorm1 = _mm_add_ps(absf, _mm_castsi128_ps(STBIR__CONSTI(c_subnorm_magic))); // magic value to round output mantissa
2438 subnorm2 = _mm_sub_epi32(_mm_castps_si128(subnorm1), STBIR__CONSTI(c_subnorm_magic)); // subtract out bias
2439
2440 // "result is normal" path
2441 mantoddbit = _mm_slli_epi32(absf_int, 31 - 13); // shift bit 13 (mantissa LSB) to sign
2442 mantodd = _mm_srai_epi32(mantoddbit, 31); // -1 if FP16 mantissa odd, else 0
2443
2444 round1 = _mm_add_epi32(absf_int, STBIR__CONSTI(c_normal_bias));
2445 round2 = _mm_sub_epi32(round1, mantodd); // if mantissa LSB odd, bias towards rounding up (RTNE)
2446 normal = _mm_srli_epi32(round2, 13); // rounded result
2447
2448 // combine the two non-specials
2449 nonspecial = _mm_or_si128(_mm_and_si128(subnorm2, b_issub), _mm_andnot_si128(b_issub, normal));
2450
2451 // merge in specials as well
2452 joined = _mm_or_si128(_mm_and_si128(nonspecial, b_isregular), _mm_andnot_si128(b_isregular, inf_or_nan));
2453
2454 sign_shift = _mm_srai_epi32(_mm_castps_si128(justsign), 16);
2455 final2 = _mm_or_si128(joined, sign_shift);
2456 final = _mm_packs_epi32(final, final2);
2457 stbir__simdi_store( output,final );
2458 }
2459
2460#elif defined(STBIR_NEON) && defined(_MSC_VER) && defined(_M_ARM64) && !defined(__clang__) // 64-bit ARM on MSVC (not clang)
2461
2462 static stbir__inline void stbir__half_to_float_SIMD(float * output, stbir__FP16 const * input)
2463 {
2464 float16x4_t in0 = vld1_f16(input + 0);
2465 float16x4_t in1 = vld1_f16(input + 4);
2466 vst1q_f32(output + 0, vcvt_f32_f16(in0));
2467 vst1q_f32(output + 4, vcvt_f32_f16(in1));
2468 }
2469
2470 static stbir__inline void stbir__float_to_half_SIMD(stbir__FP16 * output, float const * input)
2471 {
2472 float16x4_t out0 = vcvt_f16_f32(vld1q_f32(input + 0));
2473 float16x4_t out1 = vcvt_f16_f32(vld1q_f32(input + 4));
2474 vst1_f16(output+0, out0);
2475 vst1_f16(output+4, out1);
2476 }
2477
2478 static stbir__inline float stbir__half_to_float( stbir__FP16 h )
2479 {
2480 return vgetq_lane_f32(vcvt_f32_f16(vld1_dup_f16(&h)), 0);
2481 }
2482
2483 static stbir__inline stbir__FP16 stbir__float_to_half( float f )
2484 {
2485 return vget_lane_f16(vcvt_f16_f32(vdupq_n_f32(f)), 0).n16_u16[0];
2486 }
2487
2488#elif defined(STBIR_NEON) && ( defined( _M_ARM64 ) || defined( __aarch64__ ) || defined( __arm64__ ) ) // 64-bit ARM
2489
2490 static stbir__inline void stbir__half_to_float_SIMD(float * output, stbir__FP16 const * input)
2491 {
2492 float16x8_t in = vld1q_f16(input);
2493 vst1q_f32(output + 0, vcvt_f32_f16(vget_low_f16(in)));
2494 vst1q_f32(output + 4, vcvt_f32_f16(vget_high_f16(in)));
2495 }
2496
2497 static stbir__inline void stbir__float_to_half_SIMD(stbir__FP16 * output, float const * input)
2498 {
2499 float16x4_t out0 = vcvt_f16_f32(vld1q_f32(input + 0));
2500 float16x4_t out1 = vcvt_f16_f32(vld1q_f32(input + 4));
2501 vst1q_f16(output, vcombine_f16(out0, out1));
2502 }
2503
2504 static stbir__inline float stbir__half_to_float( stbir__FP16 h )
2505 {
2506 return vgetq_lane_f32(vcvt_f32_f16(vdup_n_f16(h)), 0);
2507 }
2508
2509 static stbir__inline stbir__FP16 stbir__float_to_half( float f )
2510 {
2511 return vget_lane_f16(vcvt_f16_f32(vdupq_n_f32(f)), 0);
2512 }
2513
2514#elif defined(STBIR_WASM) || (defined(STBIR_NEON) && (defined(_MSC_VER) || defined(_M_ARM) || defined(__arm__))) // WASM or 32-bit ARM on MSVC/clang
2515
2516 static stbir__inline void stbir__half_to_float_SIMD(float * output, stbir__FP16 const * input)
2517 {
2518 for (int i=0; i<8; i++)
2519 {
2520 output[i] = stbir__half_to_float(input[i]);
2521 }
2522 }
2523 static stbir__inline void stbir__float_to_half_SIMD(stbir__FP16 * output, float const * input)
2524 {
2525 for (int i=0; i<8; i++)
2526 {
2527 output[i] = stbir__float_to_half(input[i]);
2528 }
2529 }
2530
2531#endif
2532
2533
2534#ifdef STBIR_SIMD
2535
2536#define stbir__simdf_0123to3333( out, reg ) (out) = stbir__simdf_swiz( reg, 3,3,3,3 )
2537#define stbir__simdf_0123to2222( out, reg ) (out) = stbir__simdf_swiz( reg, 2,2,2,2 )
2538#define stbir__simdf_0123to1111( out, reg ) (out) = stbir__simdf_swiz( reg, 1,1,1,1 )
2539#define stbir__simdf_0123to0000( out, reg ) (out) = stbir__simdf_swiz( reg, 0,0,0,0 )
2540#define stbir__simdf_0123to0003( out, reg ) (out) = stbir__simdf_swiz( reg, 0,0,0,3 )
2541#define stbir__simdf_0123to0001( out, reg ) (out) = stbir__simdf_swiz( reg, 0,0,0,1 )
2542#define stbir__simdf_0123to1122( out, reg ) (out) = stbir__simdf_swiz( reg, 1,1,2,2 )
2543#define stbir__simdf_0123to2333( out, reg ) (out) = stbir__simdf_swiz( reg, 2,3,3,3 )
2544#define stbir__simdf_0123to0023( out, reg ) (out) = stbir__simdf_swiz( reg, 0,0,2,3 )
2545#define stbir__simdf_0123to1230( out, reg ) (out) = stbir__simdf_swiz( reg, 1,2,3,0 )
2546#define stbir__simdf_0123to2103( out, reg ) (out) = stbir__simdf_swiz( reg, 2,1,0,3 )
2547#define stbir__simdf_0123to3210( out, reg ) (out) = stbir__simdf_swiz( reg, 3,2,1,0 )
2548#define stbir__simdf_0123to2301( out, reg ) (out) = stbir__simdf_swiz( reg, 2,3,0,1 )
2549#define stbir__simdf_0123to3012( out, reg ) (out) = stbir__simdf_swiz( reg, 3,0,1,2 )
2550#define stbir__simdf_0123to0011( out, reg ) (out) = stbir__simdf_swiz( reg, 0,0,1,1 )
2551#define stbir__simdf_0123to1100( out, reg ) (out) = stbir__simdf_swiz( reg, 1,1,0,0 )
2552#define stbir__simdf_0123to2233( out, reg ) (out) = stbir__simdf_swiz( reg, 2,2,3,3 )
2553#define stbir__simdf_0123to1133( out, reg ) (out) = stbir__simdf_swiz( reg, 1,1,3,3 )
2554#define stbir__simdf_0123to0022( out, reg ) (out) = stbir__simdf_swiz( reg, 0,0,2,2 )
2555#define stbir__simdf_0123to1032( out, reg ) (out) = stbir__simdf_swiz( reg, 1,0,3,2 )
2556
2557typedef union stbir__simdi_u32
2558{
2559 stbir_uint32 m128i_u32[4];
2560 int m128i_i32[4];
2561 stbir__simdi m128i_i128;
2562} stbir__simdi_u32;
2563
2564static const int STBIR_mask[9] = { 0,0,0,-1,-1,-1,0,0,0 };
2565
2566static const STBIR__SIMDF_CONST(STBIR_max_uint8_as_float, stbir__max_uint8_as_float);
2567static const STBIR__SIMDF_CONST(STBIR_max_uint16_as_float, stbir__max_uint16_as_float);
2568static const STBIR__SIMDF_CONST(STBIR_max_uint8_as_float_inverted, stbir__max_uint8_as_float_inverted);
2569static const STBIR__SIMDF_CONST(STBIR_max_uint16_as_float_inverted, stbir__max_uint16_as_float_inverted);
2570
2571static const STBIR__SIMDF_CONST(STBIR_simd_point5, 0.5f);
2572static const STBIR__SIMDF_CONST(STBIR_ones, 1.0f);
2573static const STBIR__SIMDI_CONST(STBIR_almost_zero, (127 - 13) << 23);
2574static const STBIR__SIMDI_CONST(STBIR_almost_one, 0x3f7fffff);
2575static const STBIR__SIMDI_CONST(STBIR_mantissa_mask, 0xff);
2576static const STBIR__SIMDI_CONST(STBIR_topscale, 0x02000000);
2577
2578// Basically, in simd mode, we unroll the proper amount, and we don't want
2579// the non-simd remnant loops to be unroll because they only run a few times
2580// Adding this switch saves about 5K on clang which is Captain Unroll the 3rd.
2581#define STBIR_SIMD_STREAMOUT_PTR( star ) STBIR_STREAMOUT_PTR( star )
2582#define STBIR_SIMD_NO_UNROLL(ptr) STBIR_NO_UNROLL(ptr)
2583#define STBIR_SIMD_NO_UNROLL_LOOP_START STBIR_NO_UNROLL_LOOP_START
2584#define STBIR_SIMD_NO_UNROLL_LOOP_START_INF_FOR STBIR_NO_UNROLL_LOOP_START_INF_FOR
2585
2586#ifdef STBIR_MEMCPY
2587#undef STBIR_MEMCPY
2588#endif
2589#define STBIR_MEMCPY stbir_simd_memcpy
2590
2591// override normal use of memcpy with much simpler copy (faster and smaller with our sized copies)
2592static void stbir_simd_memcpy( void * dest, void const * src, size_t bytes )
2593{
2594 char STBIR_SIMD_STREAMOUT_PTR (*) d = (char*) dest;
2595 char STBIR_SIMD_STREAMOUT_PTR( * ) d_end = ((char*) dest) + bytes;
2596 ptrdiff_t ofs_to_src = (char*)src - (char*)dest;
2597
2598 // check overlaps
2599 STBIR_ASSERT( ( ( d >= ( (char*)src) + bytes ) ) || ( ( d + bytes ) <= (char*)src ) );
2600
2601 if ( bytes < (16*stbir__simdfX_float_count) )
2602 {
2603 if ( bytes < 16 )
2604 {
2605 if ( bytes )
2606 {
2607 STBIR_SIMD_NO_UNROLL_LOOP_START
2608 do
2609 {
2610 STBIR_SIMD_NO_UNROLL(d);
2611 d[ 0 ] = d[ ofs_to_src ];
2612 ++d;
2613 } while ( d < d_end );
2614 }
2615 }
2616 else
2617 {
2618 stbir__simdf x;
2619 // do one unaligned to get us aligned for the stream out below
2620 stbir__simdf_load( x, ( d + ofs_to_src ) );
2621 stbir__simdf_store( d, x );
2622 d = (char*)( ( ( (size_t)d ) + 16 ) & ~15 );
2623
2624 STBIR_SIMD_NO_UNROLL_LOOP_START_INF_FOR
2625 for(;;)
2626 {
2627 STBIR_SIMD_NO_UNROLL(d);
2628
2629 if ( d > ( d_end - 16 ) )
2630 {
2631 if ( d == d_end )
2632 return;
2633 d = d_end - 16;
2634 }
2635
2636 stbir__simdf_load( x, ( d + ofs_to_src ) );
2637 stbir__simdf_store( d, x );
2638 d += 16;
2639 }
2640 }
2641 }
2642 else
2643 {
2644 stbir__simdfX x0,x1,x2,x3;
2645
2646 // do one unaligned to get us aligned for the stream out below
2647 stbir__simdfX_load( x0, ( d + ofs_to_src ) + 0*stbir__simdfX_float_count );
2648 stbir__simdfX_load( x1, ( d + ofs_to_src ) + 4*stbir__simdfX_float_count );
2649 stbir__simdfX_load( x2, ( d + ofs_to_src ) + 8*stbir__simdfX_float_count );
2650 stbir__simdfX_load( x3, ( d + ofs_to_src ) + 12*stbir__simdfX_float_count );
2651 stbir__simdfX_store( d + 0*stbir__simdfX_float_count, x0 );
2652 stbir__simdfX_store( d + 4*stbir__simdfX_float_count, x1 );
2653 stbir__simdfX_store( d + 8*stbir__simdfX_float_count, x2 );
2654 stbir__simdfX_store( d + 12*stbir__simdfX_float_count, x3 );
2655 d = (char*)( ( ( (size_t)d ) + (16*stbir__simdfX_float_count) ) & ~((16*stbir__simdfX_float_count)-1) );
2656
2657 STBIR_SIMD_NO_UNROLL_LOOP_START_INF_FOR
2658 for(;;)
2659 {
2660 STBIR_SIMD_NO_UNROLL(d);
2661
2662 if ( d > ( d_end - (16*stbir__simdfX_float_count) ) )
2663 {
2664 if ( d == d_end )
2665 return;
2666 d = d_end - (16*stbir__simdfX_float_count);
2667 }
2668
2669 stbir__simdfX_load( x0, ( d + ofs_to_src ) + 0*stbir__simdfX_float_count );
2670 stbir__simdfX_load( x1, ( d + ofs_to_src ) + 4*stbir__simdfX_float_count );
2671 stbir__simdfX_load( x2, ( d + ofs_to_src ) + 8*stbir__simdfX_float_count );
2672 stbir__simdfX_load( x3, ( d + ofs_to_src ) + 12*stbir__simdfX_float_count );
2673 stbir__simdfX_store( d + 0*stbir__simdfX_float_count, x0 );
2674 stbir__simdfX_store( d + 4*stbir__simdfX_float_count, x1 );
2675 stbir__simdfX_store( d + 8*stbir__simdfX_float_count, x2 );
2676 stbir__simdfX_store( d + 12*stbir__simdfX_float_count, x3 );
2677 d += (16*stbir__simdfX_float_count);
2678 }
2679 }
2680}
2681
2682// memcpy that is specically intentionally overlapping (src is smaller then dest, so can be
2683// a normal forward copy, bytes is divisible by 4 and bytes is greater than or equal to
2684// the diff between dest and src)
2685static void stbir_overlapping_memcpy( void * dest, void const * src, size_t bytes )
2686{
2687 char STBIR_SIMD_STREAMOUT_PTR (*) sd = (char*) src;
2688 char STBIR_SIMD_STREAMOUT_PTR( * ) s_end = ((char*) src) + bytes;
2689 ptrdiff_t ofs_to_dest = (char*)dest - (char*)src;
2690
2691 if ( ofs_to_dest >= 16 ) // is the overlap more than 16 away?
2692 {
2693 char STBIR_SIMD_STREAMOUT_PTR( * ) s_end16 = ((char*) src) + (bytes&~15);
2694 STBIR_SIMD_NO_UNROLL_LOOP_START
2695 do
2696 {
2697 stbir__simdf x;
2698 STBIR_SIMD_NO_UNROLL(sd);
2699 stbir__simdf_load( x, sd );
2700 stbir__simdf_store( ( sd + ofs_to_dest ), x );
2701 sd += 16;
2702 } while ( sd < s_end16 );
2703
2704 if ( sd == s_end )
2705 return;
2706 }
2707
2708 do
2709 {
2710 STBIR_SIMD_NO_UNROLL(sd);
2711 *(int*)( sd + ofs_to_dest ) = *(int*) sd;
2712 sd += 4;
2713 } while ( sd < s_end );
2714}
2715
2716#else // no SSE2
2717
2718// when in scalar mode, we let unrolling happen, so this macro just does the __restrict
2719#define STBIR_SIMD_STREAMOUT_PTR( star ) STBIR_STREAMOUT_PTR( star )
2720#define STBIR_SIMD_NO_UNROLL(ptr)
2721#define STBIR_SIMD_NO_UNROLL_LOOP_START
2722#define STBIR_SIMD_NO_UNROLL_LOOP_START_INF_FOR
2723
2724#endif // SSE2
2725
2726
2727#ifdef STBIR_PROFILE
2728
2729#ifndef STBIR_PROFILE_FUNC
2730
2731#if defined(_x86_64) || defined( __x86_64__ ) || defined( _M_X64 ) || defined(__x86_64) || defined(__SSE2__) || defined(STBIR_SSE) || defined( _M_IX86_FP ) || defined(__i386) || defined( __i386__ ) || defined( _M_IX86 ) || defined( _X86_ )
2732
2733#ifdef _MSC_VER
2734
2735 STBIRDEF stbir_uint64 __rdtsc();
2736 #define STBIR_PROFILE_FUNC() __rdtsc()
2737
2738#else // non msvc
2739
2740 static stbir__inline stbir_uint64 STBIR_PROFILE_FUNC()
2741 {
2742 stbir_uint32 lo, hi;
2743 asm volatile ("rdtsc" : "=a" (lo), "=d" (hi) );
2744 return ( ( (stbir_uint64) hi ) << 32 ) | ( (stbir_uint64) lo );
2745 }
2746
2747#endif // msvc
2748
2749#elif defined( _M_ARM64 ) || defined( __aarch64__ ) || defined( __arm64__ ) || defined(__ARM_NEON__)
2750
2751#if defined( _MSC_VER ) && !defined(__clang__)
2752
2753 #define STBIR_PROFILE_FUNC() _ReadStatusReg(ARM64_CNTVCT)
2754
2755#else
2756
2757 static stbir__inline stbir_uint64 STBIR_PROFILE_FUNC()
2758 {
2759 stbir_uint64 tsc;
2760 asm volatile("mrs %0, cntvct_el0" : "=r" (tsc));
2761 return tsc;
2762 }
2763
2764#endif
2765
2766#else // x64, arm
2767
2768#error Unknown platform for profiling.
2769
2770#endif // x64, arm
2771
2772#endif // STBIR_PROFILE_FUNC
2773
2774#define STBIR_ONLY_PROFILE_GET_SPLIT_INFO ,stbir__per_split_info * split_info
2775#define STBIR_ONLY_PROFILE_SET_SPLIT_INFO ,split_info
2776
2777#define STBIR_ONLY_PROFILE_BUILD_GET_INFO ,stbir__info * profile_info
2778#define STBIR_ONLY_PROFILE_BUILD_SET_INFO ,profile_info
2779
2780// super light-weight micro profiler
2781#define STBIR_PROFILE_START_ll( info, wh ) { stbir_uint64 wh##thiszonetime = STBIR_PROFILE_FUNC(); stbir_uint64 * wh##save_parent_excluded_ptr = info->current_zone_excluded_ptr; stbir_uint64 wh##current_zone_excluded = 0; info->current_zone_excluded_ptr = &wh##current_zone_excluded;
2782#define STBIR_PROFILE_END_ll( info, wh ) wh##thiszonetime = STBIR_PROFILE_FUNC() - wh##thiszonetime; info->profile.named.wh += wh##thiszonetime - wh##current_zone_excluded; *wh##save_parent_excluded_ptr += wh##thiszonetime; info->current_zone_excluded_ptr = wh##save_parent_excluded_ptr; }
2783#define STBIR_PROFILE_FIRST_START_ll( info, wh ) { int i; info->current_zone_excluded_ptr = &info->profile.named.total; for(i=0;i<STBIR__ARRAY_SIZE(info->profile.array);i++) info->profile.array[i]=0; } STBIR_PROFILE_START_ll( info, wh );
2784#define STBIR_PROFILE_CLEAR_EXTRAS_ll( info, num ) { int extra; for(extra=1;extra<(num);extra++) { int i; for(i=0;i<STBIR__ARRAY_SIZE((info)->profile.array);i++) (info)[extra].profile.array[i]=0; } }
2785
2786// for thread data
2787#define STBIR_PROFILE_START( wh ) STBIR_PROFILE_START_ll( split_info, wh )
2788#define STBIR_PROFILE_END( wh ) STBIR_PROFILE_END_ll( split_info, wh )
2789#define STBIR_PROFILE_FIRST_START( wh ) STBIR_PROFILE_FIRST_START_ll( split_info, wh )
2790#define STBIR_PROFILE_CLEAR_EXTRAS() STBIR_PROFILE_CLEAR_EXTRAS_ll( split_info, split_count )
2791
2792// for build data
2793#define STBIR_PROFILE_BUILD_START( wh ) STBIR_PROFILE_START_ll( profile_info, wh )
2794#define STBIR_PROFILE_BUILD_END( wh ) STBIR_PROFILE_END_ll( profile_info, wh )
2795#define STBIR_PROFILE_BUILD_FIRST_START( wh ) STBIR_PROFILE_FIRST_START_ll( profile_info, wh )
2796#define STBIR_PROFILE_BUILD_CLEAR( info ) { int i; for(i=0;i<STBIR__ARRAY_SIZE(info->profile.array);i++) info->profile.array[i]=0; }
2797
2798#else // no profile
2799
2800#define STBIR_ONLY_PROFILE_GET_SPLIT_INFO
2801#define STBIR_ONLY_PROFILE_SET_SPLIT_INFO
2802
2803#define STBIR_ONLY_PROFILE_BUILD_GET_INFO
2804#define STBIR_ONLY_PROFILE_BUILD_SET_INFO
2805
2806#define STBIR_PROFILE_START( wh )
2807#define STBIR_PROFILE_END( wh )
2808#define STBIR_PROFILE_FIRST_START( wh )
2809#define STBIR_PROFILE_CLEAR_EXTRAS( )
2810
2811#define STBIR_PROFILE_BUILD_START( wh )
2812#define STBIR_PROFILE_BUILD_END( wh )
2813#define STBIR_PROFILE_BUILD_FIRST_START( wh )
2814#define STBIR_PROFILE_BUILD_CLEAR( info )
2815
2816#endif // stbir_profile
2817
2818#ifndef STBIR_CEILF
2819#include <math.h>
2820#if _MSC_VER <= 1200 // support VC6 for Sean
2821#define STBIR_CEILF(x) ((float)ceil((float)(x)))
2822#define STBIR_FLOORF(x) ((float)floor((float)(x)))
2823#else
2824#define STBIR_CEILF(x) ceilf(x)
2825#define STBIR_FLOORF(x) floorf(x)
2826#endif
2827#endif
2828
2829#ifndef STBIR_MEMCPY
2830// For memcpy
2831#include <string.h>
2832#define STBIR_MEMCPY( dest, src, len ) memcpy( dest, src, len )
2833#endif
2834
2835#ifndef STBIR_SIMD
2836
2837// memcpy that is specifically intentionally overlapping (src is smaller then dest, so can be
2838// a normal forward copy, bytes is divisible by 4 and bytes is greater than or equal to
2839// the diff between dest and src)
2840static void stbir_overlapping_memcpy( void * dest, void const * src, size_t bytes )
2841{
2842 char STBIR_SIMD_STREAMOUT_PTR (*) sd = (char*) src;
2843 char STBIR_SIMD_STREAMOUT_PTR( * ) s_end = ((char*) src) + bytes;
2844 ptrdiff_t ofs_to_dest = (char*)dest - (char*)src;
2845
2846 if ( ofs_to_dest >= 8 ) // is the overlap more than 8 away
2847 {
2848 char STBIR_SIMD_STREAMOUT_PTR( * ) s_end8 = ((char*) src) + (bytes&~7);
2849
2850 if ( ( ( ((ptrdiff_t)dest)|((ptrdiff_t)src) ) & 7 ) == 0 ) // is it 8byte aligned?
2851 {
2852 STBIR_NO_UNROLL_LOOP_START
2853 do
2854 {
2855 STBIR_NO_UNROLL(sd);
2856 *(stbir_uint64*)( sd + ofs_to_dest ) = *(stbir_uint64*) sd;
2857 sd += 8;
2858 } while ( sd < s_end8 );
2859 }
2860 else
2861 {
2862 STBIR_NO_UNROLL_LOOP_START
2863 do
2864 {
2865 int a,b;
2866 STBIR_NO_UNROLL(sd);
2867 a = ((int*)sd)[0];
2868 b = ((int*)sd)[1];
2869 ((int*)( sd + ofs_to_dest ))[0] = a;
2870 ((int*)( sd + ofs_to_dest ))[1] = b;
2871 sd += 8;
2872 } while ( sd < s_end8 );
2873 }
2874
2875 if ( sd == s_end )
2876 return;
2877 }
2878
2879 STBIR_NO_UNROLL_LOOP_START
2880 do
2881 {
2882 STBIR_NO_UNROLL(sd);
2883 *(int*)( sd + ofs_to_dest ) = *(int*) sd;
2884 sd += 4;
2885 } while ( sd < s_end );
2886}
2887
2888#endif
2889
2890static float stbir__filter_trapezoid(float x, float scale, void * user_data)
2891{
2892 float halfscale = scale / 2;
2893 float t = 0.5f + halfscale;
2894 STBIR_ASSERT(scale <= 1);
2895 STBIR__UNUSED(user_data);
2896
2897 if ( x < 0.0f ) x = -x;
2898
2899 if (x >= t)
2900 return 0.0f;
2901 else
2902 {
2903 float r = 0.5f - halfscale;
2904 if (x <= r)
2905 return 1.0f;
2906 else
2907 return (t - x) / scale;
2908 }
2909}
2910
2911static float stbir__support_trapezoid(float scale, void * user_data)
2912{
2913 STBIR__UNUSED(user_data);
2914 return 0.5f + scale / 2.0f;
2915}
2916
2917static float stbir__filter_triangle(float x, float s, void * user_data)
2918{
2919 STBIR__UNUSED(s);
2920 STBIR__UNUSED(user_data);
2921
2922 if ( x < 0.0f ) x = -x;
2923
2924 if (x <= 1.0f)
2925 return 1.0f - x;
2926 else
2927 return 0.0f;
2928}
2929
2930static float stbir__filter_point(float x, float s, void * user_data)
2931{
2932 STBIR__UNUSED(x);
2933 STBIR__UNUSED(s);
2934 STBIR__UNUSED(user_data);
2935
2936 return 1.0f;
2937}
2938
2939static float stbir__filter_cubic(float x, float s, void * user_data)
2940{
2941 STBIR__UNUSED(s);
2942 STBIR__UNUSED(user_data);
2943
2944 if ( x < 0.0f ) x = -x;
2945
2946 if (x < 1.0f)
2947 return (4.0f + x*x*(3.0f*x - 6.0f))/6.0f;
2948 else if (x < 2.0f)
2949 return (8.0f + x*(-12.0f + x*(6.0f - x)))/6.0f;
2950
2951 return (0.0f);
2952}
2953
2954static float stbir__filter_catmullrom(float x, float s, void * user_data)
2955{
2956 STBIR__UNUSED(s);
2957 STBIR__UNUSED(user_data);
2958
2959 if ( x < 0.0f ) x = -x;
2960
2961 if (x < 1.0f)
2962 return 1.0f - x*x*(2.5f - 1.5f*x);
2963 else if (x < 2.0f)
2964 return 2.0f - x*(4.0f + x*(0.5f*x - 2.5f));
2965
2966 return (0.0f);
2967}
2968
2969static float stbir__filter_mitchell(float x, float s, void * user_data)
2970{
2971 STBIR__UNUSED(s);
2972 STBIR__UNUSED(user_data);
2973
2974 if ( x < 0.0f ) x = -x;
2975
2976 if (x < 1.0f)
2977 return (16.0f + x*x*(21.0f * x - 36.0f))/18.0f;
2978 else if (x < 2.0f)
2979 return (32.0f + x*(-60.0f + x*(36.0f - 7.0f*x)))/18.0f;
2980
2981 return (0.0f);
2982}
2983
2984static float stbir__support_zeropoint5(float s, void * user_data)
2985{
2986 STBIR__UNUSED(s);
2987 STBIR__UNUSED(user_data);
2988 return 0.5f;
2989}
2990
2991static float stbir__support_one(float s, void * user_data)
2992{
2993 STBIR__UNUSED(s);
2994 STBIR__UNUSED(user_data);
2995 return 1;
2996}
2997
2998static float stbir__support_two(float s, void * user_data)
2999{
3000 STBIR__UNUSED(s);
3001 STBIR__UNUSED(user_data);
3002 return 2;
3003}
3004
3005// This is the maximum number of input samples that can affect an output sample
3006// with the given filter from the output pixel's perspective
3007static int stbir__get_filter_pixel_width(stbir__support_callback * support, float scale, void * user_data)
3008{
3009 STBIR_ASSERT(support != 0);
3010
3011 if ( scale >= ( 1.0f-stbir__small_float ) ) // upscale
3012 return (int)STBIR_CEILF(support(1.0f/scale,user_data) * 2.0f);
3013 else
3014 return (int)STBIR_CEILF(support(scale,user_data) * 2.0f / scale);
3015}
3016
3017// this is how many coefficents per run of the filter (which is different
3018// from the filter_pixel_width depending on if we are scattering or gathering)
3019static int stbir__get_coefficient_width(stbir__sampler * samp, int is_gather, void * user_data)
3020{
3021 float scale = samp->scale_info.scale;
3022 stbir__support_callback * support = samp->filter_support;
3023
3024 switch( is_gather )
3025 {
3026 case 1:
3027 return (int)STBIR_CEILF(support(1.0f / scale, user_data) * 2.0f);
3028 case 2:
3029 return (int)STBIR_CEILF(support(scale, user_data) * 2.0f / scale);
3030 case 0:
3031 return (int)STBIR_CEILF(support(scale, user_data) * 2.0f);
3032 default:
3033 STBIR_ASSERT( (is_gather >= 0 ) && (is_gather <= 2 ) );
3034 return 0;
3035 }
3036}
3037
3038static int stbir__get_contributors(stbir__sampler * samp, int is_gather)
3039{
3040 if (is_gather)
3041 return samp->scale_info.output_sub_size;
3042 else
3043 return (samp->scale_info.input_full_size + samp->filter_pixel_margin * 2);
3044}
3045
3046static int stbir__edge_zero_full( int n, int max )
3047{
3048 STBIR__UNUSED(n);
3049 STBIR__UNUSED(max);
3050 return 0; // NOTREACHED
3051}
3052
3053static int stbir__edge_clamp_full( int n, int max )
3054{
3055 if (n < 0)
3056 return 0;
3057
3058 if (n >= max)
3059 return max - 1;
3060
3061 return n; // NOTREACHED
3062}
3063
3064static int stbir__edge_reflect_full( int n, int max )
3065{
3066 if (n < 0)
3067 {
3068 if (n > -max)
3069 return -n;
3070 else
3071 return max - 1;
3072 }
3073
3074 if (n >= max)
3075 {
3076 int max2 = max * 2;
3077 if (n >= max2)
3078 return 0;
3079 else
3080 return max2 - n - 1;
3081 }
3082
3083 return n; // NOTREACHED
3084}
3085
3086static int stbir__edge_wrap_full( int n, int max )
3087{
3088 if (n >= 0)
3089 return (n % max);
3090 else
3091 {
3092 int m = (-n) % max;
3093
3094 if (m != 0)
3095 m = max - m;
3096
3097 return (m);
3098 }
3099}
3100
3101typedef int stbir__edge_wrap_func( int n, int max );
3102static stbir__edge_wrap_func * stbir__edge_wrap_slow[] =
3103{
3104 stbir__edge_clamp_full, // STBIR_EDGE_CLAMP
3105 stbir__edge_reflect_full, // STBIR_EDGE_REFLECT
3106 stbir__edge_wrap_full, // STBIR_EDGE_WRAP
3107 stbir__edge_zero_full, // STBIR_EDGE_ZERO
3108};
3109
3110stbir__inline static int stbir__edge_wrap(stbir_edge edge, int n, int max)
3111{
3112 // avoid per-pixel switch
3113 if (n >= 0 && n < max)
3114 return n;
3115 return stbir__edge_wrap_slow[edge]( n, max );
3116}
3117
3118#define STBIR__MERGE_RUNS_PIXEL_THRESHOLD 16
3119
3120// get information on the extents of a sampler
3121static void stbir__get_extents( stbir__sampler * samp, stbir__extents * scanline_extents )
3122{
3123 int j, stop;
3124 int left_margin, right_margin;
3125 int min_n = 0x7fffffff, max_n = -0x7fffffff;
3126 int min_left = 0x7fffffff, max_left = -0x7fffffff;
3127 int min_right = 0x7fffffff, max_right = -0x7fffffff;
3128 stbir_edge edge = samp->edge;
3129 stbir__contributors* contributors = samp->contributors;
3130 int output_sub_size = samp->scale_info.output_sub_size;
3131 int input_full_size = samp->scale_info.input_full_size;
3132 int filter_pixel_margin = samp->filter_pixel_margin;
3133
3134 STBIR_ASSERT( samp->is_gather );
3135
3136 stop = output_sub_size;
3137 for (j = 0; j < stop; j++ )
3138 {
3139 STBIR_ASSERT( contributors[j].n1 >= contributors[j].n0 );
3140 if ( contributors[j].n0 < min_n )
3141 {
3142 min_n = contributors[j].n0;
3143 stop = j + filter_pixel_margin; // if we find a new min, only scan another filter width
3144 if ( stop > output_sub_size ) stop = output_sub_size;
3145 }
3146 }
3147
3148 stop = 0;
3149 for (j = output_sub_size - 1; j >= stop; j-- )
3150 {
3151 STBIR_ASSERT( contributors[j].n1 >= contributors[j].n0 );
3152 if ( contributors[j].n1 > max_n )
3153 {
3154 max_n = contributors[j].n1;
3155 stop = j - filter_pixel_margin; // if we find a new max, only scan another filter width
3156 if (stop<0) stop = 0;
3157 }
3158 }
3159
3160 STBIR_ASSERT( scanline_extents->conservative.n0 <= min_n );
3161 STBIR_ASSERT( scanline_extents->conservative.n1 >= max_n );
3162
3163 // now calculate how much into the margins we really read
3164 left_margin = 0;
3165 if ( min_n < 0 )
3166 {
3167 left_margin = -min_n;
3168 min_n = 0;
3169 }
3170
3171 right_margin = 0;
3172 if ( max_n >= input_full_size )
3173 {
3174 right_margin = max_n - input_full_size + 1;
3175 max_n = input_full_size - 1;
3176 }
3177
3178 // index 1 is margin pixel extents (how many pixels we hang over the edge)
3179 scanline_extents->edge_sizes[0] = left_margin;
3180 scanline_extents->edge_sizes[1] = right_margin;
3181
3182 // index 2 is pixels read from the input
3183 scanline_extents->spans[0].n0 = min_n;
3184 scanline_extents->spans[0].n1 = max_n;
3185 scanline_extents->spans[0].pixel_offset_for_input = min_n;
3186
3187 // default to no other input range
3188 scanline_extents->spans[1].n0 = 0;
3189 scanline_extents->spans[1].n1 = -1;
3190 scanline_extents->spans[1].pixel_offset_for_input = 0;
3191
3192 // don't have to do edge calc for zero clamp
3193 if ( edge == STBIR_EDGE_ZERO )
3194 return;
3195
3196 // convert margin pixels to the pixels within the input (min and max)
3197 for( j = -left_margin ; j < 0 ; j++ )
3198 {
3199 int p = stbir__edge_wrap( edge, j, input_full_size );
3200 if ( p < min_left )
3201 min_left = p;
3202 if ( p > max_left )
3203 max_left = p;
3204 }
3205
3206 for( j = input_full_size ; j < (input_full_size + right_margin) ; j++ )
3207 {
3208 int p = stbir__edge_wrap( edge, j, input_full_size );
3209 if ( p < min_right )
3210 min_right = p;
3211 if ( p > max_right )
3212 max_right = p;
3213 }
3214
3215 // merge the left margin pixel region if it connects within 4 pixels of main pixel region
3216 if ( min_left != 0x7fffffff )
3217 {
3218 if ( ( ( min_left <= min_n ) && ( ( max_left + STBIR__MERGE_RUNS_PIXEL_THRESHOLD ) >= min_n ) ) ||
3219 ( ( min_n <= min_left ) && ( ( max_n + STBIR__MERGE_RUNS_PIXEL_THRESHOLD ) >= max_left ) ) )
3220 {
3221 scanline_extents->spans[0].n0 = min_n = stbir__min( min_n, min_left );
3222 scanline_extents->spans[0].n1 = max_n = stbir__max( max_n, max_left );
3223 scanline_extents->spans[0].pixel_offset_for_input = min_n;
3224 left_margin = 0;
3225 }
3226 }
3227
3228 // merge the right margin pixel region if it connects within 4 pixels of main pixel region
3229 if ( min_right != 0x7fffffff )
3230 {
3231 if ( ( ( min_right <= min_n ) && ( ( max_right + STBIR__MERGE_RUNS_PIXEL_THRESHOLD ) >= min_n ) ) ||
3232 ( ( min_n <= min_right ) && ( ( max_n + STBIR__MERGE_RUNS_PIXEL_THRESHOLD ) >= max_right ) ) )
3233 {
3234 scanline_extents->spans[0].n0 = min_n = stbir__min( min_n, min_right );
3235 scanline_extents->spans[0].n1 = max_n = stbir__max( max_n, max_right );
3236 scanline_extents->spans[0].pixel_offset_for_input = min_n;
3237 right_margin = 0;
3238 }
3239 }
3240
3241 STBIR_ASSERT( scanline_extents->conservative.n0 <= min_n );
3242 STBIR_ASSERT( scanline_extents->conservative.n1 >= max_n );
3243
3244 // you get two ranges when you have the WRAP edge mode and you are doing just the a piece of the resize
3245 // so you need to get a second run of pixels from the opposite side of the scanline (which you
3246 // wouldn't need except for WRAP)
3247
3248
3249 // if we can't merge the min_left range, add it as a second range
3250 if ( ( left_margin ) && ( min_left != 0x7fffffff ) )
3251 {
3252 stbir__span * newspan = scanline_extents->spans + 1;
3253 STBIR_ASSERT( right_margin == 0 );
3254 if ( min_left < scanline_extents->spans[0].n0 )
3255 {
3256 scanline_extents->spans[1].pixel_offset_for_input = scanline_extents->spans[0].n0;
3257 scanline_extents->spans[1].n0 = scanline_extents->spans[0].n0;
3258 scanline_extents->spans[1].n1 = scanline_extents->spans[0].n1;
3259 --newspan;
3260 }
3261 newspan->pixel_offset_for_input = min_left;
3262 newspan->n0 = -left_margin;
3263 newspan->n1 = ( max_left - min_left ) - left_margin;
3264 scanline_extents->edge_sizes[0] = 0; // don't need to copy the left margin, since we are directly decoding into the margin
3265 }
3266 // if we can't merge the min_right range, add it as a second range
3267 else
3268 if ( ( right_margin ) && ( min_right != 0x7fffffff ) )
3269 {
3270 stbir__span * newspan = scanline_extents->spans + 1;
3271 if ( min_right < scanline_extents->spans[0].n0 )
3272 {
3273 scanline_extents->spans[1].pixel_offset_for_input = scanline_extents->spans[0].n0;
3274 scanline_extents->spans[1].n0 = scanline_extents->spans[0].n0;
3275 scanline_extents->spans[1].n1 = scanline_extents->spans[0].n1;
3276 --newspan;
3277 }
3278 newspan->pixel_offset_for_input = min_right;
3279 newspan->n0 = scanline_extents->spans[1].n1 + 1;
3280 newspan->n1 = scanline_extents->spans[1].n1 + 1 + ( max_right - min_right );
3281 scanline_extents->edge_sizes[1] = 0; // don't need to copy the right margin, since we are directly decoding into the margin
3282 }
3283
3284 // sort the spans into write output order
3285 if ( ( scanline_extents->spans[1].n1 > scanline_extents->spans[1].n0 ) && ( scanline_extents->spans[0].n0 > scanline_extents->spans[1].n0 ) )
3286 {
3287 stbir__span tspan = scanline_extents->spans[0];
3288 scanline_extents->spans[0] = scanline_extents->spans[1];
3289 scanline_extents->spans[1] = tspan;
3290 }
3291}
3292
3293static void stbir__calculate_in_pixel_range( int * first_pixel, int * last_pixel, float out_pixel_center, float out_filter_radius, float inv_scale, float out_shift, int input_size, stbir_edge edge )
3294{
3295 int first, last;
3296 float out_pixel_influence_lowerbound = out_pixel_center - out_filter_radius;
3297 float out_pixel_influence_upperbound = out_pixel_center + out_filter_radius;
3298
3299 float in_pixel_influence_lowerbound = (out_pixel_influence_lowerbound + out_shift) * inv_scale;
3300 float in_pixel_influence_upperbound = (out_pixel_influence_upperbound + out_shift) * inv_scale;
3301
3302 first = (int)(STBIR_FLOORF(in_pixel_influence_lowerbound + 0.5f));
3303 last = (int)(STBIR_FLOORF(in_pixel_influence_upperbound - 0.5f));
3304 if ( last < first ) last = first; // point sample mode can span a value *right* at 0.5, and cause these to cross
3305
3306 if ( edge == STBIR_EDGE_WRAP )
3307 {
3308 if ( first < -input_size )
3309 first = -input_size;
3310 if ( last >= (input_size*2))
3311 last = (input_size*2) - 1;
3312 }
3313
3314 *first_pixel = first;
3315 *last_pixel = last;
3316}
3317
3318static void stbir__calculate_coefficients_for_gather_upsample( float out_filter_radius, stbir__kernel_callback * kernel, stbir__scale_info * scale_info, int num_contributors, stbir__contributors* contributors, float* coefficient_group, int coefficient_width, stbir_edge edge, void * user_data )
3319{
3320 int n, end;
3321 float inv_scale = scale_info->inv_scale;
3322 float out_shift = scale_info->pixel_shift;
3323 int input_size = scale_info->input_full_size;
3324 int numerator = scale_info->scale_numerator;
3325 int polyphase = ( ( scale_info->scale_is_rational ) && ( numerator < num_contributors ) );
3326
3327 // Looping through out pixels
3328 end = num_contributors; if ( polyphase ) end = numerator;
3329 for (n = 0; n < end; n++)
3330 {
3331 int i;
3332 int last_non_zero;
3333 float out_pixel_center = (float)n + 0.5f;
3334 float in_center_of_out = (out_pixel_center + out_shift) * inv_scale;
3335
3336 int in_first_pixel, in_last_pixel;
3337
3338 stbir__calculate_in_pixel_range( &in_first_pixel, &in_last_pixel, out_pixel_center, out_filter_radius, inv_scale, out_shift, input_size, edge );
3339
3340 // make sure we never generate a range larger than our precalculated coeff width
3341 // this only happens in point sample mode, but it's a good safe thing to do anyway
3342 if ( ( in_last_pixel - in_first_pixel + 1 ) > coefficient_width )
3343 in_last_pixel = in_first_pixel + coefficient_width - 1;
3344
3345 last_non_zero = -1;
3346 for (i = 0; i <= in_last_pixel - in_first_pixel; i++)
3347 {
3348 float in_pixel_center = (float)(i + in_first_pixel) + 0.5f;
3349 float coeff = kernel(in_center_of_out - in_pixel_center, inv_scale, user_data);
3350
3351 // kill denormals
3352 if ( ( ( coeff < stbir__small_float ) && ( coeff > -stbir__small_float ) ) )
3353 {
3354 if ( i == 0 ) // if we're at the front, just eat zero contributors
3355 {
3356 STBIR_ASSERT ( ( in_last_pixel - in_first_pixel ) != 0 ); // there should be at least one contrib
3357 ++in_first_pixel;
3358 i--;
3359 continue;
3360 }
3361 coeff = 0; // make sure is fully zero (should keep denormals away)
3362 }
3363 else
3364 last_non_zero = i;
3365
3366 coefficient_group[i] = coeff;
3367 }
3368
3369 in_last_pixel = last_non_zero+in_first_pixel; // kills trailing zeros
3370 contributors->n0 = in_first_pixel;
3371 contributors->n1 = in_last_pixel;
3372
3373 STBIR_ASSERT(contributors->n1 >= contributors->n0);
3374
3375 ++contributors;
3376 coefficient_group += coefficient_width;
3377 }
3378}
3379
3380static void stbir__insert_coeff( stbir__contributors * contribs, float * coeffs, int new_pixel, float new_coeff, int max_width )
3381{
3382 if ( contribs->n1 < contribs->n0 ) // this first clause should never happen, but handle in case
3383 {
3384 contribs->n0 = contribs->n1 = new_pixel;
3385 coeffs[0] = new_coeff;
3386 }
3387 else if ( new_pixel <= contribs->n1 ) // before the end
3388 {
3389 if ( new_pixel < contribs->n0 ) // before the front?
3390 {
3391 if ( ( contribs->n1 - new_pixel + 1 ) <= max_width )
3392 {
3393 int j, o = contribs->n0 - new_pixel;
3394 for ( j = contribs->n1 - contribs->n0 ; j >= 0 ; j-- )
3395 coeffs[ j + o ] = coeffs[ j ];
3396 for ( j = 1 ; j < o ; j++ )
3397 coeffs[ j ] = 0;
3398 coeffs[ 0 ] = new_coeff;
3399 contribs->n0 = new_pixel;
3400 }
3401 }
3402 else
3403 {
3404 // add new weight to existing coeff if already there
3405 coeffs[ new_pixel - contribs->n0 ] += new_coeff;
3406 }
3407 }
3408 else
3409 {
3410 if ( ( new_pixel - contribs->n0 + 1 ) <= max_width )
3411 {
3412 int j, e = new_pixel - contribs->n0;
3413 for( j = ( contribs->n1 - contribs->n0 ) + 1 ; j < e ; j++ ) // clear in-betweens coeffs if there are any
3414 coeffs[j] = 0;
3415
3416 coeffs[ e ] = new_coeff;
3417 contribs->n1 = new_pixel;
3418 }
3419 }
3420}
3421
3422static void stbir__calculate_out_pixel_range( int * first_pixel, int * last_pixel, float in_pixel_center, float in_pixels_radius, float scale, float out_shift, int out_size )
3423{
3424 float in_pixel_influence_lowerbound = in_pixel_center - in_pixels_radius;
3425 float in_pixel_influence_upperbound = in_pixel_center + in_pixels_radius;
3426 float out_pixel_influence_lowerbound = in_pixel_influence_lowerbound * scale - out_shift;
3427 float out_pixel_influence_upperbound = in_pixel_influence_upperbound * scale - out_shift;
3428 int out_first_pixel = (int)(STBIR_FLOORF(out_pixel_influence_lowerbound + 0.5f));
3429 int out_last_pixel = (int)(STBIR_FLOORF(out_pixel_influence_upperbound - 0.5f));
3430
3431 if ( out_first_pixel < 0 )
3432 out_first_pixel = 0;
3433 if ( out_last_pixel >= out_size )
3434 out_last_pixel = out_size - 1;
3435 *first_pixel = out_first_pixel;
3436 *last_pixel = out_last_pixel;
3437}
3438
3439static void stbir__calculate_coefficients_for_gather_downsample( int start, int end, float in_pixels_radius, stbir__kernel_callback * kernel, stbir__scale_info * scale_info, int coefficient_width, int num_contributors, stbir__contributors * contributors, float * coefficient_group, void * user_data )
3440{
3441 int in_pixel;
3442 int i;
3443 int first_out_inited = -1;
3444 float scale = scale_info->scale;
3445 float out_shift = scale_info->pixel_shift;
3446 int out_size = scale_info->output_sub_size;
3447 int numerator = scale_info->scale_numerator;
3448 int polyphase = ( ( scale_info->scale_is_rational ) && ( numerator < out_size ) );
3449
3450 STBIR__UNUSED(num_contributors);
3451
3452 // Loop through the input pixels
3453 for (in_pixel = start; in_pixel < end; in_pixel++)
3454 {
3455 float in_pixel_center = (float)in_pixel + 0.5f;
3456 float out_center_of_in = in_pixel_center * scale - out_shift;
3457 int out_first_pixel, out_last_pixel;
3458
3459 stbir__calculate_out_pixel_range( &out_first_pixel, &out_last_pixel, in_pixel_center, in_pixels_radius, scale, out_shift, out_size );
3460
3461 if ( out_first_pixel > out_last_pixel )
3462 continue;
3463
3464 // clamp or exit if we are using polyphase filtering, and the limit is up
3465 if ( polyphase )
3466 {
3467 // when polyphase, you only have to do coeffs up to the numerator count
3468 if ( out_first_pixel == numerator )
3469 break;
3470
3471 // don't do any extra work, clamp last pixel at numerator too
3472 if ( out_last_pixel >= numerator )
3473 out_last_pixel = numerator - 1;
3474 }
3475
3476 for (i = 0; i <= out_last_pixel - out_first_pixel; i++)
3477 {
3478 float out_pixel_center = (float)(i + out_first_pixel) + 0.5f;
3479 float x = out_pixel_center - out_center_of_in;
3480 float coeff = kernel(x, scale, user_data) * scale;
3481
3482 // kill the coeff if it's too small (avoid denormals)
3483 if ( ( ( coeff < stbir__small_float ) && ( coeff > -stbir__small_float ) ) )
3484 coeff = 0.0f;
3485
3486 {
3487 int out = i + out_first_pixel;
3488 float * coeffs = coefficient_group + out * coefficient_width;
3489 stbir__contributors * contribs = contributors + out;
3490
3491 // is this the first time this output pixel has been seen? Init it.
3492 if ( out > first_out_inited )
3493 {
3494 STBIR_ASSERT( out == ( first_out_inited + 1 ) ); // ensure we have only advanced one at time
3495 first_out_inited = out;
3496 contribs->n0 = in_pixel;
3497 contribs->n1 = in_pixel;
3498 coeffs[0] = coeff;
3499 }
3500 else
3501 {
3502 // insert on end (always in order)
3503 if ( coeffs[0] == 0.0f ) // if the first coefficent is zero, then zap it for this coeffs
3504 {
3505 STBIR_ASSERT( ( in_pixel - contribs->n0 ) == 1 ); // ensure that when we zap, we're at the 2nd pos
3506 contribs->n0 = in_pixel;
3507 }
3508 contribs->n1 = in_pixel;
3509 STBIR_ASSERT( ( in_pixel - contribs->n0 ) < coefficient_width );
3510 coeffs[in_pixel - contribs->n0] = coeff;
3511 }
3512 }
3513 }
3514 }
3515}
3516
3517#ifdef STBIR_RENORMALIZE_IN_FLOAT
3518#define STBIR_RENORM_TYPE float
3519#else
3520#define STBIR_RENORM_TYPE double
3521#endif
3522
3523static void stbir__cleanup_gathered_coefficients( stbir_edge edge, stbir__filter_extent_info* filter_info, stbir__scale_info * scale_info, int num_contributors, stbir__contributors* contributors, float * coefficient_group, int coefficient_width )
3524{
3525 int input_size = scale_info->input_full_size;
3526 int input_last_n1 = input_size - 1;
3527 int n, end;
3528 int lowest = 0x7fffffff;
3529 int highest = -0x7fffffff;
3530 int widest = -1;
3531 int numerator = scale_info->scale_numerator;
3532 int denominator = scale_info->scale_denominator;
3533 int polyphase = ( ( scale_info->scale_is_rational ) && ( numerator < num_contributors ) );
3534 float * coeffs;
3535 stbir__contributors * contribs;
3536
3537 // weight all the coeffs for each sample
3538 coeffs = coefficient_group;
3539 contribs = contributors;
3540 end = num_contributors; if ( polyphase ) end = numerator;
3541 for (n = 0; n < end; n++)
3542 {
3543 int i;
3544 STBIR_RENORM_TYPE filter_scale, total_filter = 0;
3545 int e;
3546
3547 // add all contribs
3548 e = contribs->n1 - contribs->n0;
3549 for( i = 0 ; i <= e ; i++ )
3550 {
3551 total_filter += (STBIR_RENORM_TYPE) coeffs[i];
3552 STBIR_ASSERT( ( coeffs[i] >= -2.0f ) && ( coeffs[i] <= 2.0f ) ); // check for wonky weights
3553 }
3554
3555 // rescale
3556 if ( ( total_filter < stbir__small_float ) && ( total_filter > -stbir__small_float ) )
3557 {
3558 // all coeffs are extremely small, just zero it
3559 contribs->n1 = contribs->n0;
3560 coeffs[0] = 0.0f;
3561 }
3562 else
3563 {
3564 // if the total isn't 1.0, rescale everything
3565 if ( ( total_filter < (1.0f-stbir__small_float) ) || ( total_filter > (1.0f+stbir__small_float) ) )
3566 {
3567 filter_scale = ((STBIR_RENORM_TYPE)1.0) / total_filter;
3568
3569 // scale them all
3570 for (i = 0; i <= e; i++)
3571 coeffs[i] = (float) ( coeffs[i] * filter_scale );
3572 }
3573 }
3574 ++contribs;
3575 coeffs += coefficient_width;
3576 }
3577
3578 // if we have a rational for the scale, we can exploit the polyphaseness to not calculate
3579 // most of the coefficients, so we copy them here
3580 if ( polyphase )
3581 {
3582 stbir__contributors * prev_contribs = contributors;
3583 stbir__contributors * cur_contribs = contributors + numerator;
3584
3585 for( n = numerator ; n < num_contributors ; n++ )
3586 {
3587 cur_contribs->n0 = prev_contribs->n0 + denominator;
3588 cur_contribs->n1 = prev_contribs->n1 + denominator;
3589 ++cur_contribs;
3590 ++prev_contribs;
3591 }
3592 stbir_overlapping_memcpy( coefficient_group + numerator * coefficient_width, coefficient_group, ( num_contributors - numerator ) * coefficient_width * sizeof( coeffs[ 0 ] ) );
3593 }
3594
3595 coeffs = coefficient_group;
3596 contribs = contributors;
3597
3598 for (n = 0; n < num_contributors; n++)
3599 {
3600 int i;
3601
3602 // in zero edge mode, just remove out of bounds contribs completely (since their weights are accounted for now)
3603 if ( edge == STBIR_EDGE_ZERO )
3604 {
3605 // shrink the right side if necessary
3606 if ( contribs->n1 > input_last_n1 )
3607 contribs->n1 = input_last_n1;
3608
3609 // shrink the left side
3610 if ( contribs->n0 < 0 )
3611 {
3612 int j, left, skips = 0;
3613
3614 skips = -contribs->n0;
3615 contribs->n0 = 0;
3616
3617 // now move down the weights
3618 left = contribs->n1 - contribs->n0 + 1;
3619 if ( left > 0 )
3620 {
3621 for( j = 0 ; j < left ; j++ )
3622 coeffs[ j ] = coeffs[ j + skips ];
3623 }
3624 }
3625 }
3626 else if ( ( edge == STBIR_EDGE_CLAMP ) || ( edge == STBIR_EDGE_REFLECT ) )
3627 {
3628 // for clamp and reflect, calculate the true inbounds position (based on edge type) and just add that to the existing weight
3629
3630 // right hand side first
3631 if ( contribs->n1 > input_last_n1 )
3632 {
3633 int start = contribs->n0;
3634 int endi = contribs->n1;
3635 contribs->n1 = input_last_n1;
3636 for( i = input_size; i <= endi; i++ )
3637 stbir__insert_coeff( contribs, coeffs, stbir__edge_wrap_slow[edge]( i, input_size ), coeffs[i-start], coefficient_width );
3638 }
3639
3640 // now check left hand edge
3641 if ( contribs->n0 < 0 )
3642 {
3643 int save_n0;
3644 float save_n0_coeff;
3645 float * c = coeffs - ( contribs->n0 + 1 );
3646
3647 // reinsert the coeffs with it reflected or clamped (insert accumulates, if the coeffs exist)
3648 for( i = -1 ; i > contribs->n0 ; i-- )
3649 stbir__insert_coeff( contribs, coeffs, stbir__edge_wrap_slow[edge]( i, input_size ), *c--, coefficient_width );
3650 save_n0 = contribs->n0;
3651 save_n0_coeff = c[0]; // save it, since we didn't do the final one (i==n0), because there might be too many coeffs to hold (before we resize)!
3652
3653 // now slide all the coeffs down (since we have accumulated them in the positive contribs) and reset the first contrib
3654 contribs->n0 = 0;
3655 for(i = 0 ; i <= contribs->n1 ; i++ )
3656 coeffs[i] = coeffs[i-save_n0];
3657
3658 // now that we have shrunk down the contribs, we insert the first one safely
3659 stbir__insert_coeff( contribs, coeffs, stbir__edge_wrap_slow[edge]( save_n0, input_size ), save_n0_coeff, coefficient_width );
3660 }
3661 }
3662
3663 if ( contribs->n0 <= contribs->n1 )
3664 {
3665 int diff = contribs->n1 - contribs->n0 + 1;
3666 while ( diff && ( coeffs[ diff-1 ] == 0.0f ) )
3667 --diff;
3668
3669 contribs->n1 = contribs->n0 + diff - 1;
3670
3671 if ( contribs->n0 <= contribs->n1 )
3672 {
3673 if ( contribs->n0 < lowest )
3674 lowest = contribs->n0;
3675 if ( contribs->n1 > highest )
3676 highest = contribs->n1;
3677 if ( diff > widest )
3678 widest = diff;
3679 }
3680
3681 // re-zero out unused coefficients (if any)
3682 for( i = diff ; i < coefficient_width ; i++ )
3683 coeffs[i] = 0.0f;
3684 }
3685
3686 ++contribs;
3687 coeffs += coefficient_width;
3688 }
3689 filter_info->lowest = lowest;
3690 filter_info->highest = highest;
3691 filter_info->widest = widest;
3692}
3693
3694#undef STBIR_RENORM_TYPE
3695
3696static int stbir__pack_coefficients( int num_contributors, stbir__contributors* contributors, float * coefficents, int coefficient_width, int widest, int row0, int row1 )
3697{
3698 #define STBIR_MOVE_1( dest, src ) { STBIR_NO_UNROLL(dest); ((stbir_uint32*)(dest))[0] = ((stbir_uint32*)(src))[0]; }
3699 #define STBIR_MOVE_2( dest, src ) { STBIR_NO_UNROLL(dest); ((stbir_uint64*)(dest))[0] = ((stbir_uint64*)(src))[0]; }
3700 #ifdef STBIR_SIMD
3701 #define STBIR_MOVE_4( dest, src ) { stbir__simdf t; STBIR_NO_UNROLL(dest); stbir__simdf_load( t, src ); stbir__simdf_store( dest, t ); }
3702 #else
3703 #define STBIR_MOVE_4( dest, src ) { STBIR_NO_UNROLL(dest); ((stbir_uint64*)(dest))[0] = ((stbir_uint64*)(src))[0]; ((stbir_uint64*)(dest))[1] = ((stbir_uint64*)(src))[1]; }
3704 #endif
3705
3706 int row_end = row1 + 1;
3707 STBIR__UNUSED( row0 ); // only used in an assert
3708
3709 if ( coefficient_width != widest )
3710 {
3711 float * pc = coefficents;
3712 float * coeffs = coefficents;
3713 float * pc_end = coefficents + num_contributors * widest;
3714 switch( widest )
3715 {
3716 case 1:
3717 STBIR_NO_UNROLL_LOOP_START
3718 do {
3719 STBIR_MOVE_1( pc, coeffs );
3720 ++pc;
3721 coeffs += coefficient_width;
3722 } while ( pc < pc_end );
3723 break;
3724 case 2:
3725 STBIR_NO_UNROLL_LOOP_START
3726 do {
3727 STBIR_MOVE_2( pc, coeffs );
3728 pc += 2;
3729 coeffs += coefficient_width;
3730 } while ( pc < pc_end );
3731 break;
3732 case 3:
3733 STBIR_NO_UNROLL_LOOP_START
3734 do {
3735 STBIR_MOVE_2( pc, coeffs );
3736 STBIR_MOVE_1( pc+2, coeffs+2 );
3737 pc += 3;
3738 coeffs += coefficient_width;
3739 } while ( pc < pc_end );
3740 break;
3741 case 4:
3742 STBIR_NO_UNROLL_LOOP_START
3743 do {
3744 STBIR_MOVE_4( pc, coeffs );
3745 pc += 4;
3746 coeffs += coefficient_width;
3747 } while ( pc < pc_end );
3748 break;
3749 case 5:
3750 STBIR_NO_UNROLL_LOOP_START
3751 do {
3752 STBIR_MOVE_4( pc, coeffs );
3753 STBIR_MOVE_1( pc+4, coeffs+4 );
3754 pc += 5;
3755 coeffs += coefficient_width;
3756 } while ( pc < pc_end );
3757 break;
3758 case 6:
3759 STBIR_NO_UNROLL_LOOP_START
3760 do {
3761 STBIR_MOVE_4( pc, coeffs );
3762 STBIR_MOVE_2( pc+4, coeffs+4 );
3763 pc += 6;
3764 coeffs += coefficient_width;
3765 } while ( pc < pc_end );
3766 break;
3767 case 7:
3768 STBIR_NO_UNROLL_LOOP_START
3769 do {
3770 STBIR_MOVE_4( pc, coeffs );
3771 STBIR_MOVE_2( pc+4, coeffs+4 );
3772 STBIR_MOVE_1( pc+6, coeffs+6 );
3773 pc += 7;
3774 coeffs += coefficient_width;
3775 } while ( pc < pc_end );
3776 break;
3777 case 8:
3778 STBIR_NO_UNROLL_LOOP_START
3779 do {
3780 STBIR_MOVE_4( pc, coeffs );
3781 STBIR_MOVE_4( pc+4, coeffs+4 );
3782 pc += 8;
3783 coeffs += coefficient_width;
3784 } while ( pc < pc_end );
3785 break;
3786 case 9:
3787 STBIR_NO_UNROLL_LOOP_START
3788 do {
3789 STBIR_MOVE_4( pc, coeffs );
3790 STBIR_MOVE_4( pc+4, coeffs+4 );
3791 STBIR_MOVE_1( pc+8, coeffs+8 );
3792 pc += 9;
3793 coeffs += coefficient_width;
3794 } while ( pc < pc_end );
3795 break;
3796 case 10:
3797 STBIR_NO_UNROLL_LOOP_START
3798 do {
3799 STBIR_MOVE_4( pc, coeffs );
3800 STBIR_MOVE_4( pc+4, coeffs+4 );
3801 STBIR_MOVE_2( pc+8, coeffs+8 );
3802 pc += 10;
3803 coeffs += coefficient_width;
3804 } while ( pc < pc_end );
3805 break;
3806 case 11:
3807 STBIR_NO_UNROLL_LOOP_START
3808 do {
3809 STBIR_MOVE_4( pc, coeffs );
3810 STBIR_MOVE_4( pc+4, coeffs+4 );
3811 STBIR_MOVE_2( pc+8, coeffs+8 );
3812 STBIR_MOVE_1( pc+10, coeffs+10 );
3813 pc += 11;
3814 coeffs += coefficient_width;
3815 } while ( pc < pc_end );
3816 break;
3817 case 12:
3818 STBIR_NO_UNROLL_LOOP_START
3819 do {
3820 STBIR_MOVE_4( pc, coeffs );
3821 STBIR_MOVE_4( pc+4, coeffs+4 );
3822 STBIR_MOVE_4( pc+8, coeffs+8 );
3823 pc += 12;
3824 coeffs += coefficient_width;
3825 } while ( pc < pc_end );
3826 break;
3827 default:
3828 STBIR_NO_UNROLL_LOOP_START
3829 do {
3830 float * copy_end = pc + widest - 4;
3831 float * c = coeffs;
3832 do {
3833 STBIR_NO_UNROLL( pc );
3834 STBIR_MOVE_4( pc, c );
3835 pc += 4;
3836 c += 4;
3837 } while ( pc <= copy_end );
3838 copy_end += 4;
3839 STBIR_NO_UNROLL_LOOP_START
3840 while ( pc < copy_end )
3841 {
3842 STBIR_MOVE_1( pc, c );
3843 ++pc; ++c;
3844 }
3845 coeffs += coefficient_width;
3846 } while ( pc < pc_end );
3847 break;
3848 }
3849 }
3850
3851 // some horizontal routines read one float off the end (which is then masked off), so put in a sentinel so we don't read an snan or denormal
3852 coefficents[ widest * num_contributors ] = 8888.0f;
3853
3854 // the minimum we might read for unrolled filters widths is 12. So, we need to
3855 // make sure we never read outside the decode buffer, by possibly moving
3856 // the sample area back into the scanline, and putting zeros weights first.
3857 // we start on the right edge and check until we're well past the possible
3858 // clip area (2*widest).
3859 {
3860 stbir__contributors * contribs = contributors + num_contributors - 1;
3861 float * coeffs = coefficents + widest * ( num_contributors - 1 );
3862
3863 // go until no chance of clipping (this is usually less than 8 lops)
3864 while ( ( contribs >= contributors ) && ( ( contribs->n0 + widest*2 ) >= row_end ) )
3865 {
3866 // might we clip??
3867 if ( ( contribs->n0 + widest ) > row_end )
3868 {
3869 int stop_range = widest;
3870
3871 // if range is larger than 12, it will be handled by generic loops that can terminate on the exact length
3872 // of this contrib n1, instead of a fixed widest amount - so calculate this
3873 if ( widest > 12 )
3874 {
3875 int mod;
3876
3877 // how far will be read in the n_coeff loop (which depends on the widest count mod4);
3878 mod = widest & 3;
3879 stop_range = ( ( ( contribs->n1 - contribs->n0 + 1 ) - mod + 3 ) & ~3 ) + mod;
3880
3881 // the n_coeff loops do a minimum amount of coeffs, so factor that in!
3882 if ( stop_range < ( 8 + mod ) ) stop_range = 8 + mod;
3883 }
3884
3885 // now see if we still clip with the refined range
3886 if ( ( contribs->n0 + stop_range ) > row_end )
3887 {
3888 int new_n0 = row_end - stop_range;
3889 int num = contribs->n1 - contribs->n0 + 1;
3890 int backup = contribs->n0 - new_n0;
3891 float * from_co = coeffs + num - 1;
3892 float * to_co = from_co + backup;
3893
3894 STBIR_ASSERT( ( new_n0 >= row0 ) && ( new_n0 < contribs->n0 ) );
3895
3896 // move the coeffs over
3897 while( num )
3898 {
3899 *to_co-- = *from_co--;
3900 --num;
3901 }
3902 // zero new positions
3903 while ( to_co >= coeffs )
3904 *to_co-- = 0;
3905 // set new start point
3906 contribs->n0 = new_n0;
3907 if ( widest > 12 )
3908 {
3909 int mod;
3910
3911 // how far will be read in the n_coeff loop (which depends on the widest count mod4);
3912 mod = widest & 3;
3913 stop_range = ( ( ( contribs->n1 - contribs->n0 + 1 ) - mod + 3 ) & ~3 ) + mod;
3914
3915 // the n_coeff loops do a minimum amount of coeffs, so factor that in!
3916 if ( stop_range < ( 8 + mod ) ) stop_range = 8 + mod;
3917 }
3918 }
3919 }
3920 --contribs;
3921 coeffs -= widest;
3922 }
3923 }
3924
3925 return widest;
3926 #undef STBIR_MOVE_1
3927 #undef STBIR_MOVE_2
3928 #undef STBIR_MOVE_4
3929}
3930
3931static void stbir__calculate_filters( stbir__sampler * samp, stbir__sampler * other_axis_for_pivot, void * user_data STBIR_ONLY_PROFILE_BUILD_GET_INFO )
3932{
3933 int n;
3934 float scale = samp->scale_info.scale;
3935 stbir__kernel_callback * kernel = samp->filter_kernel;
3936 stbir__support_callback * support = samp->filter_support;
3937 float inv_scale = samp->scale_info.inv_scale;
3938 int input_full_size = samp->scale_info.input_full_size;
3939 int gather_num_contributors = samp->num_contributors;
3940 stbir__contributors* gather_contributors = samp->contributors;
3941 float * gather_coeffs = samp->coefficients;
3942 int gather_coefficient_width = samp->coefficient_width;
3943
3944 switch ( samp->is_gather )
3945 {
3946 case 1: // gather upsample
3947 {
3948 float out_pixels_radius = support(inv_scale,user_data) * scale;
3949
3950 stbir__calculate_coefficients_for_gather_upsample( out_pixels_radius, kernel, &samp->scale_info, gather_num_contributors, gather_contributors, gather_coeffs, gather_coefficient_width, samp->edge, user_data );
3951
3952 STBIR_PROFILE_BUILD_START( cleanup );
3953 stbir__cleanup_gathered_coefficients( samp->edge, &samp->extent_info, &samp->scale_info, gather_num_contributors, gather_contributors, gather_coeffs, gather_coefficient_width );
3954 STBIR_PROFILE_BUILD_END( cleanup );
3955 }
3956 break;
3957
3958 case 0: // scatter downsample (only on vertical)
3959 case 2: // gather downsample
3960 {
3961 float in_pixels_radius = support(scale,user_data) * inv_scale;
3962 int filter_pixel_margin = samp->filter_pixel_margin;
3963 int input_end = input_full_size + filter_pixel_margin;
3964
3965 // if this is a scatter, we do a downsample gather to get the coeffs, and then pivot after
3966 if ( !samp->is_gather )
3967 {
3968 // check if we are using the same gather downsample on the horizontal as this vertical,
3969 // if so, then we don't have to generate them, we can just pivot from the horizontal.
3970 if ( other_axis_for_pivot )
3971 {
3972 gather_contributors = other_axis_for_pivot->contributors;
3973 gather_coeffs = other_axis_for_pivot->coefficients;
3974 gather_coefficient_width = other_axis_for_pivot->coefficient_width;
3975 gather_num_contributors = other_axis_for_pivot->num_contributors;
3976 samp->extent_info.lowest = other_axis_for_pivot->extent_info.lowest;
3977 samp->extent_info.highest = other_axis_for_pivot->extent_info.highest;
3978 samp->extent_info.widest = other_axis_for_pivot->extent_info.widest;
3979 goto jump_right_to_pivot;
3980 }
3981
3982 gather_contributors = samp->gather_prescatter_contributors;
3983 gather_coeffs = samp->gather_prescatter_coefficients;
3984 gather_coefficient_width = samp->gather_prescatter_coefficient_width;
3985 gather_num_contributors = samp->gather_prescatter_num_contributors;
3986 }
3987
3988 stbir__calculate_coefficients_for_gather_downsample( -filter_pixel_margin, input_end, in_pixels_radius, kernel, &samp->scale_info, gather_coefficient_width, gather_num_contributors, gather_contributors, gather_coeffs, user_data );
3989
3990 STBIR_PROFILE_BUILD_START( cleanup );
3991 stbir__cleanup_gathered_coefficients( samp->edge, &samp->extent_info, &samp->scale_info, gather_num_contributors, gather_contributors, gather_coeffs, gather_coefficient_width );
3992 STBIR_PROFILE_BUILD_END( cleanup );
3993
3994 if ( !samp->is_gather )
3995 {
3996 // if this is a scatter (vertical only), then we need to pivot the coeffs
3997 stbir__contributors * scatter_contributors;
3998 int highest_set;
3999
4000 jump_right_to_pivot:
4001
4002 STBIR_PROFILE_BUILD_START( pivot );
4003
4004 highest_set = (-filter_pixel_margin) - 1;
4005 for (n = 0; n < gather_num_contributors; n++)
4006 {
4007 int k;
4008 int gn0 = gather_contributors->n0, gn1 = gather_contributors->n1;
4009 int scatter_coefficient_width = samp->coefficient_width;
4010 float * scatter_coeffs = samp->coefficients + ( gn0 + filter_pixel_margin ) * scatter_coefficient_width;
4011 float * g_coeffs = gather_coeffs;
4012 scatter_contributors = samp->contributors + ( gn0 + filter_pixel_margin );
4013
4014 for (k = gn0 ; k <= gn1 ; k++ )
4015 {
4016 float gc = *g_coeffs++;
4017
4018 // skip zero and denormals - must skip zeros to avoid adding coeffs beyond scatter_coefficient_width
4019 // (which happens when pivoting from horizontal, which might have dummy zeros)
4020 if ( ( ( gc >= stbir__small_float ) || ( gc <= -stbir__small_float ) ) )
4021 {
4022 if ( ( k > highest_set ) || ( scatter_contributors->n0 > scatter_contributors->n1 ) )
4023 {
4024 {
4025 // if we are skipping over several contributors, we need to clear the skipped ones
4026 stbir__contributors * clear_contributors = samp->contributors + ( highest_set + filter_pixel_margin + 1);
4027 while ( clear_contributors < scatter_contributors )
4028 {
4029 clear_contributors->n0 = 0;
4030 clear_contributors->n1 = -1;
4031 ++clear_contributors;
4032 }
4033 }
4034 scatter_contributors->n0 = n;
4035 scatter_contributors->n1 = n;
4036 scatter_coeffs[0] = gc;
4037 highest_set = k;
4038 }
4039 else
4040 {
4041 stbir__insert_coeff( scatter_contributors, scatter_coeffs, n, gc, scatter_coefficient_width );
4042 }
4043 STBIR_ASSERT( ( scatter_contributors->n1 - scatter_contributors->n0 + 1 ) <= scatter_coefficient_width );
4044 }
4045 ++scatter_contributors;
4046 scatter_coeffs += scatter_coefficient_width;
4047 }
4048
4049 ++gather_contributors;
4050 gather_coeffs += gather_coefficient_width;
4051 }
4052
4053 // now clear any unset contribs
4054 {
4055 stbir__contributors * clear_contributors = samp->contributors + ( highest_set + filter_pixel_margin + 1);
4056 stbir__contributors * end_contributors = samp->contributors + samp->num_contributors;
4057 while ( clear_contributors < end_contributors )
4058 {
4059 clear_contributors->n0 = 0;
4060 clear_contributors->n1 = -1;
4061 ++clear_contributors;
4062 }
4063 }
4064
4065 STBIR_PROFILE_BUILD_END( pivot );
4066 }
4067 }
4068 break;
4069 }
4070}
4071
4072
4073//========================================================================================================
4074// scanline decoders and encoders
4075
4076#define stbir__coder_min_num 1
4077#define STB_IMAGE_RESIZE_DO_CODERS
4078#include STBIR__HEADER_FILENAME
4079
4080#define stbir__decode_suffix BGRA
4081#define stbir__decode_swizzle
4082#define stbir__decode_order0 2
4083#define stbir__decode_order1 1
4084#define stbir__decode_order2 0
4085#define stbir__decode_order3 3
4086#define stbir__encode_order0 2
4087#define stbir__encode_order1 1
4088#define stbir__encode_order2 0
4089#define stbir__encode_order3 3
4090#define stbir__coder_min_num 4
4091#define STB_IMAGE_RESIZE_DO_CODERS
4092#include STBIR__HEADER_FILENAME
4093
4094#define stbir__decode_suffix ARGB
4095#define stbir__decode_swizzle
4096#define stbir__decode_order0 1
4097#define stbir__decode_order1 2
4098#define stbir__decode_order2 3
4099#define stbir__decode_order3 0
4100#define stbir__encode_order0 3
4101#define stbir__encode_order1 0
4102#define stbir__encode_order2 1
4103#define stbir__encode_order3 2
4104#define stbir__coder_min_num 4
4105#define STB_IMAGE_RESIZE_DO_CODERS
4106#include STBIR__HEADER_FILENAME
4107
4108#define stbir__decode_suffix ABGR
4109#define stbir__decode_swizzle
4110#define stbir__decode_order0 3
4111#define stbir__decode_order1 2
4112#define stbir__decode_order2 1
4113#define stbir__decode_order3 0
4114#define stbir__encode_order0 3
4115#define stbir__encode_order1 2
4116#define stbir__encode_order2 1
4117#define stbir__encode_order3 0
4118#define stbir__coder_min_num 4
4119#define STB_IMAGE_RESIZE_DO_CODERS
4120#include STBIR__HEADER_FILENAME
4121
4122#define stbir__decode_suffix AR
4123#define stbir__decode_swizzle
4124#define stbir__decode_order0 1
4125#define stbir__decode_order1 0
4126#define stbir__decode_order2 3
4127#define stbir__decode_order3 2
4128#define stbir__encode_order0 1
4129#define stbir__encode_order1 0
4130#define stbir__encode_order2 3
4131#define stbir__encode_order3 2
4132#define stbir__coder_min_num 2
4133#define STB_IMAGE_RESIZE_DO_CODERS
4134#include STBIR__HEADER_FILENAME
4135
4136
4137// fancy alpha means we expand to keep both premultipied and non-premultiplied color channels
4138static void stbir__fancy_alpha_weight_4ch( float * out_buffer, int width_times_channels )
4139{
4140 float STBIR_STREAMOUT_PTR(*) out = out_buffer;
4141 float const * end_decode = out_buffer + ( width_times_channels / 4 ) * 7; // decode buffer aligned to end of out_buffer
4142 float STBIR_STREAMOUT_PTR(*) decode = (float*)end_decode - width_times_channels;
4143
4144 // fancy alpha is stored internally as R G B A Rpm Gpm Bpm
4145
4146 #ifdef STBIR_SIMD
4147
4148 #ifdef STBIR_SIMD8
4149 decode += 16;
4150 STBIR_NO_UNROLL_LOOP_START
4151 while ( decode <= end_decode )
4152 {
4153 stbir__simdf8 d0,d1,a0,a1,p0,p1;
4154 STBIR_NO_UNROLL(decode);
4155 stbir__simdf8_load( d0, decode-16 );
4156 stbir__simdf8_load( d1, decode-16+8 );
4157 stbir__simdf8_0123to33333333( a0, d0 );
4158 stbir__simdf8_0123to33333333( a1, d1 );
4159 stbir__simdf8_mult( p0, a0, d0 );
4160 stbir__simdf8_mult( p1, a1, d1 );
4161 stbir__simdf8_bot4s( a0, d0, p0 );
4162 stbir__simdf8_bot4s( a1, d1, p1 );
4163 stbir__simdf8_top4s( d0, d0, p0 );
4164 stbir__simdf8_top4s( d1, d1, p1 );
4165 stbir__simdf8_store ( out, a0 );
4166 stbir__simdf8_store ( out+7, d0 );
4167 stbir__simdf8_store ( out+14, a1 );
4168 stbir__simdf8_store ( out+21, d1 );
4169 decode += 16;
4170 out += 28;
4171 }
4172 decode -= 16;
4173 #else
4174 decode += 8;
4175 STBIR_NO_UNROLL_LOOP_START
4176 while ( decode <= end_decode )
4177 {
4178 stbir__simdf d0,a0,d1,a1,p0,p1;
4179 STBIR_NO_UNROLL(decode);
4180 stbir__simdf_load( d0, decode-8 );
4181 stbir__simdf_load( d1, decode-8+4 );
4182 stbir__simdf_0123to3333( a0, d0 );
4183 stbir__simdf_0123to3333( a1, d1 );
4184 stbir__simdf_mult( p0, a0, d0 );
4185 stbir__simdf_mult( p1, a1, d1 );
4186 stbir__simdf_store ( out, d0 );
4187 stbir__simdf_store ( out+4, p0 );
4188 stbir__simdf_store ( out+7, d1 );
4189 stbir__simdf_store ( out+7+4, p1 );
4190 decode += 8;
4191 out += 14;
4192 }
4193 decode -= 8;
4194 #endif
4195
4196 // might be one last odd pixel
4197 #ifdef STBIR_SIMD8
4198 STBIR_NO_UNROLL_LOOP_START
4199 while ( decode < end_decode )
4200 #else
4201 if ( decode < end_decode )
4202 #endif
4203 {
4204 stbir__simdf d,a,p;
4205 STBIR_NO_UNROLL(decode);
4206 stbir__simdf_load( d, decode );
4207 stbir__simdf_0123to3333( a, d );
4208 stbir__simdf_mult( p, a, d );
4209 stbir__simdf_store ( out, d );
4210 stbir__simdf_store ( out+4, p );
4211 decode += 4;
4212 out += 7;
4213 }
4214
4215 #else
4216
4217 while( decode < end_decode )
4218 {
4219 float r = decode[0], g = decode[1], b = decode[2], alpha = decode[3];
4220 out[0] = r;
4221 out[1] = g;
4222 out[2] = b;
4223 out[3] = alpha;
4224 out[4] = r * alpha;
4225 out[5] = g * alpha;
4226 out[6] = b * alpha;
4227 out += 7;
4228 decode += 4;
4229 }
4230
4231 #endif
4232}
4233
4234static void stbir__fancy_alpha_weight_2ch( float * out_buffer, int width_times_channels )
4235{
4236 float STBIR_STREAMOUT_PTR(*) out = out_buffer;
4237 float const * end_decode = out_buffer + ( width_times_channels / 2 ) * 3;
4238 float STBIR_STREAMOUT_PTR(*) decode = (float*)end_decode - width_times_channels;
4239
4240 // for fancy alpha, turns into: [X A Xpm][X A Xpm],etc
4241
4242 #ifdef STBIR_SIMD
4243
4244 decode += 8;
4245 if ( decode <= end_decode )
4246 {
4247 STBIR_NO_UNROLL_LOOP_START
4248 do {
4249 #ifdef STBIR_SIMD8
4250 stbir__simdf8 d0,a0,p0;
4251 STBIR_NO_UNROLL(decode);
4252 stbir__simdf8_load( d0, decode-8 );
4253 stbir__simdf8_0123to11331133( p0, d0 );
4254 stbir__simdf8_0123to00220022( a0, d0 );
4255 stbir__simdf8_mult( p0, p0, a0 );
4256
4257 stbir__simdf_store2( out, stbir__if_simdf8_cast_to_simdf4( d0 ) );
4258 stbir__simdf_store( out+2, stbir__if_simdf8_cast_to_simdf4( p0 ) );
4259 stbir__simdf_store2h( out+3, stbir__if_simdf8_cast_to_simdf4( d0 ) );
4260
4261 stbir__simdf_store2( out+6, stbir__simdf8_gettop4( d0 ) );
4262 stbir__simdf_store( out+8, stbir__simdf8_gettop4( p0 ) );
4263 stbir__simdf_store2h( out+9, stbir__simdf8_gettop4( d0 ) );
4264 #else
4265 stbir__simdf d0,a0,d1,a1,p0,p1;
4266 STBIR_NO_UNROLL(decode);
4267 stbir__simdf_load( d0, decode-8 );
4268 stbir__simdf_load( d1, decode-8+4 );
4269 stbir__simdf_0123to1133( p0, d0 );
4270 stbir__simdf_0123to1133( p1, d1 );
4271 stbir__simdf_0123to0022( a0, d0 );
4272 stbir__simdf_0123to0022( a1, d1 );
4273 stbir__simdf_mult( p0, p0, a0 );
4274 stbir__simdf_mult( p1, p1, a1 );
4275
4276 stbir__simdf_store2( out, d0 );
4277 stbir__simdf_store( out+2, p0 );
4278 stbir__simdf_store2h( out+3, d0 );
4279
4280 stbir__simdf_store2( out+6, d1 );
4281 stbir__simdf_store( out+8, p1 );
4282 stbir__simdf_store2h( out+9, d1 );
4283 #endif
4284 decode += 8;
4285 out += 12;
4286 } while ( decode <= end_decode );
4287 }
4288 decode -= 8;
4289 #endif
4290
4291 STBIR_SIMD_NO_UNROLL_LOOP_START
4292 while( decode < end_decode )
4293 {
4294 float x = decode[0], y = decode[1];
4295 STBIR_SIMD_NO_UNROLL(decode);
4296 out[0] = x;
4297 out[1] = y;
4298 out[2] = x * y;
4299 out += 3;
4300 decode += 2;
4301 }
4302}
4303
4304static void stbir__fancy_alpha_unweight_4ch( float * encode_buffer, int width_times_channels )
4305{
4306 float STBIR_SIMD_STREAMOUT_PTR(*) encode = encode_buffer;
4307 float STBIR_SIMD_STREAMOUT_PTR(*) input = encode_buffer;
4308 float const * end_output = encode_buffer + width_times_channels;
4309
4310 // fancy RGBA is stored internally as R G B A Rpm Gpm Bpm
4311
4312 STBIR_SIMD_NO_UNROLL_LOOP_START
4313 do {
4314 float alpha = input[3];
4315#ifdef STBIR_SIMD
4316 stbir__simdf i,ia;
4317 STBIR_SIMD_NO_UNROLL(encode);
4318 if ( alpha < stbir__small_float )
4319 {
4320 stbir__simdf_load( i, input );
4321 stbir__simdf_store( encode, i );
4322 }
4323 else
4324 {
4325 stbir__simdf_load1frep4( ia, 1.0f / alpha );
4326 stbir__simdf_load( i, input+4 );
4327 stbir__simdf_mult( i, i, ia );
4328 stbir__simdf_store( encode, i );
4329 encode[3] = alpha;
4330 }
4331#else
4332 if ( alpha < stbir__small_float )
4333 {
4334 encode[0] = input[0];
4335 encode[1] = input[1];
4336 encode[2] = input[2];
4337 }
4338 else
4339 {
4340 float ialpha = 1.0f / alpha;
4341 encode[0] = input[4] * ialpha;
4342 encode[1] = input[5] * ialpha;
4343 encode[2] = input[6] * ialpha;
4344 }
4345 encode[3] = alpha;
4346#endif
4347
4348 input += 7;
4349 encode += 4;
4350 } while ( encode < end_output );
4351}
4352
4353// format: [X A Xpm][X A Xpm] etc
4354static void stbir__fancy_alpha_unweight_2ch( float * encode_buffer, int width_times_channels )
4355{
4356 float STBIR_SIMD_STREAMOUT_PTR(*) encode = encode_buffer;
4357 float STBIR_SIMD_STREAMOUT_PTR(*) input = encode_buffer;
4358 float const * end_output = encode_buffer + width_times_channels;
4359
4360 do {
4361 float alpha = input[1];
4362 encode[0] = input[0];
4363 if ( alpha >= stbir__small_float )
4364 encode[0] = input[2] / alpha;
4365 encode[1] = alpha;
4366
4367 input += 3;
4368 encode += 2;
4369 } while ( encode < end_output );
4370}
4371
4372static void stbir__simple_alpha_weight_4ch( float * decode_buffer, int width_times_channels )
4373{
4374 float STBIR_STREAMOUT_PTR(*) decode = decode_buffer;
4375 float const * end_decode = decode_buffer + width_times_channels;
4376
4377 #ifdef STBIR_SIMD
4378 {
4379 decode += 2 * stbir__simdfX_float_count;
4380 STBIR_NO_UNROLL_LOOP_START
4381 while ( decode <= end_decode )
4382 {
4383 stbir__simdfX d0,a0,d1,a1;
4384 STBIR_NO_UNROLL(decode);
4385 stbir__simdfX_load( d0, decode-2*stbir__simdfX_float_count );
4386 stbir__simdfX_load( d1, decode-2*stbir__simdfX_float_count+stbir__simdfX_float_count );
4387 stbir__simdfX_aaa1( a0, d0, STBIR_onesX );
4388 stbir__simdfX_aaa1( a1, d1, STBIR_onesX );
4389 stbir__simdfX_mult( d0, d0, a0 );
4390 stbir__simdfX_mult( d1, d1, a1 );
4391 stbir__simdfX_store ( decode-2*stbir__simdfX_float_count, d0 );
4392 stbir__simdfX_store ( decode-2*stbir__simdfX_float_count+stbir__simdfX_float_count, d1 );
4393 decode += 2 * stbir__simdfX_float_count;
4394 }
4395 decode -= 2 * stbir__simdfX_float_count;
4396
4397 // few last pixels remnants
4398 #ifdef STBIR_SIMD8
4399 STBIR_NO_UNROLL_LOOP_START
4400 while ( decode < end_decode )
4401 #else
4402 if ( decode < end_decode )
4403 #endif
4404 {
4405 stbir__simdf d,a;
4406 stbir__simdf_load( d, decode );
4407 stbir__simdf_aaa1( a, d, STBIR__CONSTF(STBIR_ones) );
4408 stbir__simdf_mult( d, d, a );
4409 stbir__simdf_store ( decode, d );
4410 decode += 4;
4411 }
4412 }
4413
4414 #else
4415
4416 while( decode < end_decode )
4417 {
4418 float alpha = decode[3];
4419 decode[0] *= alpha;
4420 decode[1] *= alpha;
4421 decode[2] *= alpha;
4422 decode += 4;
4423 }
4424
4425 #endif
4426}
4427
4428static void stbir__simple_alpha_weight_2ch( float * decode_buffer, int width_times_channels )
4429{
4430 float STBIR_STREAMOUT_PTR(*) decode = decode_buffer;
4431 float const * end_decode = decode_buffer + width_times_channels;
4432
4433 #ifdef STBIR_SIMD
4434 decode += 2 * stbir__simdfX_float_count;
4435 STBIR_NO_UNROLL_LOOP_START
4436 while ( decode <= end_decode )
4437 {
4438 stbir__simdfX d0,a0,d1,a1;
4439 STBIR_NO_UNROLL(decode);
4440 stbir__simdfX_load( d0, decode-2*stbir__simdfX_float_count );
4441 stbir__simdfX_load( d1, decode-2*stbir__simdfX_float_count+stbir__simdfX_float_count );
4442 stbir__simdfX_a1a1( a0, d0, STBIR_onesX );
4443 stbir__simdfX_a1a1( a1, d1, STBIR_onesX );
4444 stbir__simdfX_mult( d0, d0, a0 );
4445 stbir__simdfX_mult( d1, d1, a1 );
4446 stbir__simdfX_store ( decode-2*stbir__simdfX_float_count, d0 );
4447 stbir__simdfX_store ( decode-2*stbir__simdfX_float_count+stbir__simdfX_float_count, d1 );
4448 decode += 2 * stbir__simdfX_float_count;
4449 }
4450 decode -= 2 * stbir__simdfX_float_count;
4451 #endif
4452
4453 STBIR_SIMD_NO_UNROLL_LOOP_START
4454 while( decode < end_decode )
4455 {
4456 float alpha = decode[1];
4457 STBIR_SIMD_NO_UNROLL(decode);
4458 decode[0] *= alpha;
4459 decode += 2;
4460 }
4461}
4462
4463static void stbir__simple_alpha_unweight_4ch( float * encode_buffer, int width_times_channels )
4464{
4465 float STBIR_SIMD_STREAMOUT_PTR(*) encode = encode_buffer;
4466 float const * end_output = encode_buffer + width_times_channels;
4467
4468 STBIR_SIMD_NO_UNROLL_LOOP_START
4469 do {
4470 float alpha = encode[3];
4471
4472#ifdef STBIR_SIMD
4473 stbir__simdf i,ia;
4474 STBIR_SIMD_NO_UNROLL(encode);
4475 if ( alpha >= stbir__small_float )
4476 {
4477 stbir__simdf_load1frep4( ia, 1.0f / alpha );
4478 stbir__simdf_load( i, encode );
4479 stbir__simdf_mult( i, i, ia );
4480 stbir__simdf_store( encode, i );
4481 encode[3] = alpha;
4482 }
4483#else
4484 if ( alpha >= stbir__small_float )
4485 {
4486 float ialpha = 1.0f / alpha;
4487 encode[0] *= ialpha;
4488 encode[1] *= ialpha;
4489 encode[2] *= ialpha;
4490 }
4491#endif
4492 encode += 4;
4493 } while ( encode < end_output );
4494}
4495
4496static void stbir__simple_alpha_unweight_2ch( float * encode_buffer, int width_times_channels )
4497{
4498 float STBIR_SIMD_STREAMOUT_PTR(*) encode = encode_buffer;
4499 float const * end_output = encode_buffer + width_times_channels;
4500
4501 do {
4502 float alpha = encode[1];
4503 if ( alpha >= stbir__small_float )
4504 encode[0] /= alpha;
4505 encode += 2;
4506 } while ( encode < end_output );
4507}
4508
4509
4510// only used in RGB->BGR or BGR->RGB
4511static void stbir__simple_flip_3ch( float * decode_buffer, int width_times_channels )
4512{
4513 float STBIR_STREAMOUT_PTR(*) decode = decode_buffer;
4514 float const * end_decode = decode_buffer + width_times_channels;
4515
4516#ifdef STBIR_SIMD
4517 #ifdef stbir__simdf_swiz2 // do we have two argument swizzles?
4518 end_decode -= 12;
4519 STBIR_NO_UNROLL_LOOP_START
4520 while( decode <= end_decode )
4521 {
4522 // on arm64 8 instructions, no overlapping stores
4523 stbir__simdf a,b,c,na,nb;
4524 STBIR_SIMD_NO_UNROLL(decode);
4525 stbir__simdf_load( a, decode );
4526 stbir__simdf_load( b, decode+4 );
4527 stbir__simdf_load( c, decode+8 );
4528
4529 na = stbir__simdf_swiz2( a, b, 2, 1, 0, 5 );
4530 b = stbir__simdf_swiz2( a, b, 4, 3, 6, 7 );
4531 nb = stbir__simdf_swiz2( b, c, 0, 1, 4, 3 );
4532 c = stbir__simdf_swiz2( b, c, 2, 7, 6, 5 );
4533
4534 stbir__simdf_store( decode, na );
4535 stbir__simdf_store( decode+4, nb );
4536 stbir__simdf_store( decode+8, c );
4537 decode += 12;
4538 }
4539 end_decode += 12;
4540 #else
4541 end_decode -= 24;
4542 STBIR_NO_UNROLL_LOOP_START
4543 while( decode <= end_decode )
4544 {
4545 // 26 instructions on x64
4546 stbir__simdf a,b,c,d,e,f,g;
4547 float i21, i23;
4548 STBIR_SIMD_NO_UNROLL(decode);
4549 stbir__simdf_load( a, decode );
4550 stbir__simdf_load( b, decode+3 );
4551 stbir__simdf_load( c, decode+6 );
4552 stbir__simdf_load( d, decode+9 );
4553 stbir__simdf_load( e, decode+12 );
4554 stbir__simdf_load( f, decode+15 );
4555 stbir__simdf_load( g, decode+18 );
4556
4557 a = stbir__simdf_swiz( a, 2, 1, 0, 3 );
4558 b = stbir__simdf_swiz( b, 2, 1, 0, 3 );
4559 c = stbir__simdf_swiz( c, 2, 1, 0, 3 );
4560 d = stbir__simdf_swiz( d, 2, 1, 0, 3 );
4561 e = stbir__simdf_swiz( e, 2, 1, 0, 3 );
4562 f = stbir__simdf_swiz( f, 2, 1, 0, 3 );
4563 g = stbir__simdf_swiz( g, 2, 1, 0, 3 );
4564
4565 // stores overlap, need to be in order,
4566 stbir__simdf_store( decode, a );
4567 i21 = decode[21];
4568 stbir__simdf_store( decode+3, b );
4569 i23 = decode[23];
4570 stbir__simdf_store( decode+6, c );
4571 stbir__simdf_store( decode+9, d );
4572 stbir__simdf_store( decode+12, e );
4573 stbir__simdf_store( decode+15, f );
4574 stbir__simdf_store( decode+18, g );
4575 decode[21] = i23;
4576 decode[23] = i21;
4577 decode += 24;
4578 }
4579 end_decode += 24;
4580 #endif
4581#else
4582 end_decode -= 12;
4583 STBIR_NO_UNROLL_LOOP_START
4584 while( decode <= end_decode )
4585 {
4586 // 16 instructions
4587 float t0,t1,t2,t3;
4588 STBIR_NO_UNROLL(decode);
4589 t0 = decode[0]; t1 = decode[3]; t2 = decode[6]; t3 = decode[9];
4590 decode[0] = decode[2]; decode[3] = decode[5]; decode[6] = decode[8]; decode[9] = decode[11];
4591 decode[2] = t0; decode[5] = t1; decode[8] = t2; decode[11] = t3;
4592 decode += 12;
4593 }
4594 end_decode += 12;
4595#endif
4596
4597 STBIR_NO_UNROLL_LOOP_START
4598 while( decode < end_decode )
4599 {
4600 float t = decode[0];
4601 STBIR_NO_UNROLL(decode);
4602 decode[0] = decode[2];
4603 decode[2] = t;
4604 decode += 3;
4605 }
4606}
4607
4608
4609
4610static void stbir__decode_scanline(stbir__info const * stbir_info, int n, float * output_buffer STBIR_ONLY_PROFILE_GET_SPLIT_INFO )
4611{
4612 int channels = stbir_info->channels;
4613 int effective_channels = stbir_info->effective_channels;
4614 int input_sample_in_bytes = stbir__type_size[stbir_info->input_type] * channels;
4615 stbir_edge edge_horizontal = stbir_info->horizontal.edge;
4616 stbir_edge edge_vertical = stbir_info->vertical.edge;
4617 int row = stbir__edge_wrap(edge_vertical, n, stbir_info->vertical.scale_info.input_full_size);
4618 const void* input_plane_data = ( (char *) stbir_info->input_data ) + (size_t)row * (size_t) stbir_info->input_stride_bytes;
4619 stbir__span const * spans = stbir_info->scanline_extents.spans;
4620 float * full_decode_buffer = output_buffer - stbir_info->scanline_extents.conservative.n0 * effective_channels;
4621 float * last_decoded = 0;
4622
4623 // if we are on edge_zero, and we get in here with an out of bounds n, then the calculate filters has failed
4624 STBIR_ASSERT( !(edge_vertical == STBIR_EDGE_ZERO && (n < 0 || n >= stbir_info->vertical.scale_info.input_full_size)) );
4625
4626 do
4627 {
4628 float * decode_buffer;
4629 void const * input_data;
4630 float * end_decode;
4631 int width_times_channels;
4632 int width;
4633
4634 if ( spans->n1 < spans->n0 )
4635 break;
4636
4637 width = spans->n1 + 1 - spans->n0;
4638 decode_buffer = full_decode_buffer + spans->n0 * effective_channels;
4639 end_decode = full_decode_buffer + ( spans->n1 + 1 ) * effective_channels;
4640 width_times_channels = width * channels;
4641
4642 // read directly out of input plane by default
4643 input_data = ( (char*)input_plane_data ) + spans->pixel_offset_for_input * input_sample_in_bytes;
4644
4645 // if we have an input callback, call it to get the input data
4646 if ( stbir_info->in_pixels_cb )
4647 {
4648 // call the callback with a temp buffer (that they can choose to use or not). the temp is just right aligned memory in the decode_buffer itself
4649 input_data = stbir_info->in_pixels_cb( ( (char*) end_decode ) - ( width * input_sample_in_bytes ) + ( ( stbir_info->input_type != STBIR_TYPE_FLOAT ) ? ( sizeof(float)*STBIR_INPUT_CALLBACK_PADDING ) : 0 ), input_plane_data, width, spans->pixel_offset_for_input, row, stbir_info->user_data );
4650 }
4651
4652 STBIR_PROFILE_START( decode );
4653 // convert the pixels info the float decode_buffer, (we index from end_decode, so that when channels<effective_channels, we are right justified in the buffer)
4654 last_decoded = stbir_info->decode_pixels( (float*)end_decode - width_times_channels, width_times_channels, input_data );
4655 STBIR_PROFILE_END( decode );
4656
4657 if (stbir_info->alpha_weight)
4658 {
4659 STBIR_PROFILE_START( alpha );
4660 stbir_info->alpha_weight( decode_buffer, width_times_channels );
4661 STBIR_PROFILE_END( alpha );
4662 }
4663
4664 ++spans;
4665 } while ( spans <= ( &stbir_info->scanline_extents.spans[1] ) );
4666
4667 // handle the edge_wrap filter (all other types are handled back out at the calculate_filter stage)
4668 // basically the idea here is that if we have the whole scanline in memory, we don't redecode the
4669 // wrapped edge pixels, and instead just memcpy them from the scanline into the edge positions
4670 if ( ( edge_horizontal == STBIR_EDGE_WRAP ) && ( stbir_info->scanline_extents.edge_sizes[0] | stbir_info->scanline_extents.edge_sizes[1] ) )
4671 {
4672 // this code only runs if we're in edge_wrap, and we're doing the entire scanline
4673 int e, start_x[2];
4674 int input_full_size = stbir_info->horizontal.scale_info.input_full_size;
4675
4676 start_x[0] = -stbir_info->scanline_extents.edge_sizes[0]; // left edge start x
4677 start_x[1] = input_full_size; // right edge
4678
4679 for( e = 0; e < 2 ; e++ )
4680 {
4681 // do each margin
4682 int margin = stbir_info->scanline_extents.edge_sizes[e];
4683 if ( margin )
4684 {
4685 int x = start_x[e];
4686 float * marg = full_decode_buffer + x * effective_channels;
4687 float const * src = full_decode_buffer + stbir__edge_wrap(edge_horizontal, x, input_full_size) * effective_channels;
4688 STBIR_MEMCPY( marg, src, margin * effective_channels * sizeof(float) );
4689 if ( e == 1 ) last_decoded = marg + margin * effective_channels;
4690 }
4691 }
4692 }
4693
4694 // some of the horizontal gathers read one float off the edge (which is masked out), but we force a zero here to make sure no NaNs leak in
4695 // (we can't pre-zero it, because the input callback can use that area as padding)
4696 last_decoded[0] = 0.0f;
4697
4698 // we clear this extra float, because the final output pixel filter kernel might have used one less coeff than the max filter width
4699 // when this happens, we do read that pixel from the input, so it too could be Nan, so just zero an extra one.
4700 // this fits because each scanline is padded by three floats (STBIR_INPUT_CALLBACK_PADDING)
4701 last_decoded[1] = 0.0f;
4702}
4703
4704
4705//=================
4706// Do 1 channel horizontal routines
4707
4708#ifdef STBIR_SIMD
4709
4710#define stbir__1_coeff_only() \
4711 stbir__simdf tot,c; \
4712 STBIR_SIMD_NO_UNROLL(decode); \
4713 stbir__simdf_load1( c, hc ); \
4714 stbir__simdf_mult1_mem( tot, c, decode );
4715
4716#define stbir__2_coeff_only() \
4717 stbir__simdf tot,c,d; \
4718 STBIR_SIMD_NO_UNROLL(decode); \
4719 stbir__simdf_load2z( c, hc ); \
4720 stbir__simdf_load2( d, decode ); \
4721 stbir__simdf_mult( tot, c, d ); \
4722 stbir__simdf_0123to1230( c, tot ); \
4723 stbir__simdf_add1( tot, tot, c );
4724
4725#define stbir__3_coeff_only() \
4726 stbir__simdf tot,c,t; \
4727 STBIR_SIMD_NO_UNROLL(decode); \
4728 stbir__simdf_load( c, hc ); \
4729 stbir__simdf_mult_mem( tot, c, decode ); \
4730 stbir__simdf_0123to1230( c, tot ); \
4731 stbir__simdf_0123to2301( t, tot ); \
4732 stbir__simdf_add1( tot, tot, c ); \
4733 stbir__simdf_add1( tot, tot, t );
4734
4735#define stbir__store_output_tiny() \
4736 stbir__simdf_store1( output, tot ); \
4737 horizontal_coefficients += coefficient_width; \
4738 ++horizontal_contributors; \
4739 output += 1;
4740
4741#define stbir__4_coeff_start() \
4742 stbir__simdf tot,c; \
4743 STBIR_SIMD_NO_UNROLL(decode); \
4744 stbir__simdf_load( c, hc ); \
4745 stbir__simdf_mult_mem( tot, c, decode ); \
4746
4747#define stbir__4_coeff_continue_from_4( ofs ) \
4748 STBIR_SIMD_NO_UNROLL(decode); \
4749 stbir__simdf_load( c, hc + (ofs) ); \
4750 stbir__simdf_madd_mem( tot, tot, c, decode+(ofs) );
4751
4752#define stbir__1_coeff_remnant( ofs ) \
4753 { stbir__simdf d; \
4754 stbir__simdf_load1z( c, hc + (ofs) ); \
4755 stbir__simdf_load1( d, decode + (ofs) ); \
4756 stbir__simdf_madd( tot, tot, d, c ); }
4757
4758#define stbir__2_coeff_remnant( ofs ) \
4759 { stbir__simdf d; \
4760 stbir__simdf_load2z( c, hc+(ofs) ); \
4761 stbir__simdf_load2( d, decode+(ofs) ); \
4762 stbir__simdf_madd( tot, tot, d, c ); }
4763
4764#define stbir__3_coeff_setup() \
4765 stbir__simdf mask; \
4766 stbir__simdf_load( mask, STBIR_mask + 3 );
4767
4768#define stbir__3_coeff_remnant( ofs ) \
4769 stbir__simdf_load( c, hc+(ofs) ); \
4770 stbir__simdf_and( c, c, mask ); \
4771 stbir__simdf_madd_mem( tot, tot, c, decode+(ofs) );
4772
4773#define stbir__store_output() \
4774 stbir__simdf_0123to2301( c, tot ); \
4775 stbir__simdf_add( tot, tot, c ); \
4776 stbir__simdf_0123to1230( c, tot ); \
4777 stbir__simdf_add1( tot, tot, c ); \
4778 stbir__simdf_store1( output, tot ); \
4779 horizontal_coefficients += coefficient_width; \
4780 ++horizontal_contributors; \
4781 output += 1;
4782
4783#else
4784
4785#define stbir__1_coeff_only() \
4786 float tot; \
4787 tot = decode[0]*hc[0];
4788
4789#define stbir__2_coeff_only() \
4790 float tot; \
4791 tot = decode[0] * hc[0]; \
4792 tot += decode[1] * hc[1];
4793
4794#define stbir__3_coeff_only() \
4795 float tot; \
4796 tot = decode[0] * hc[0]; \
4797 tot += decode[1] * hc[1]; \
4798 tot += decode[2] * hc[2];
4799
4800#define stbir__store_output_tiny() \
4801 output[0] = tot; \
4802 horizontal_coefficients += coefficient_width; \
4803 ++horizontal_contributors; \
4804 output += 1;
4805
4806#define stbir__4_coeff_start() \
4807 float tot0,tot1,tot2,tot3; \
4808 tot0 = decode[0] * hc[0]; \
4809 tot1 = decode[1] * hc[1]; \
4810 tot2 = decode[2] * hc[2]; \
4811 tot3 = decode[3] * hc[3];
4812
4813#define stbir__4_coeff_continue_from_4( ofs ) \
4814 tot0 += decode[0+(ofs)] * hc[0+(ofs)]; \
4815 tot1 += decode[1+(ofs)] * hc[1+(ofs)]; \
4816 tot2 += decode[2+(ofs)] * hc[2+(ofs)]; \
4817 tot3 += decode[3+(ofs)] * hc[3+(ofs)];
4818
4819#define stbir__1_coeff_remnant( ofs ) \
4820 tot0 += decode[0+(ofs)] * hc[0+(ofs)];
4821
4822#define stbir__2_coeff_remnant( ofs ) \
4823 tot0 += decode[0+(ofs)] * hc[0+(ofs)]; \
4824 tot1 += decode[1+(ofs)] * hc[1+(ofs)]; \
4825
4826#define stbir__3_coeff_remnant( ofs ) \
4827 tot0 += decode[0+(ofs)] * hc[0+(ofs)]; \
4828 tot1 += decode[1+(ofs)] * hc[1+(ofs)]; \
4829 tot2 += decode[2+(ofs)] * hc[2+(ofs)];
4830
4831#define stbir__store_output() \
4832 output[0] = (tot0+tot2)+(tot1+tot3); \
4833 horizontal_coefficients += coefficient_width; \
4834 ++horizontal_contributors; \
4835 output += 1;
4836
4837#endif
4838
4839#define STBIR__horizontal_channels 1
4840#define STB_IMAGE_RESIZE_DO_HORIZONTALS
4841#include STBIR__HEADER_FILENAME
4842
4843
4844//=================
4845// Do 2 channel horizontal routines
4846
4847#ifdef STBIR_SIMD
4848
4849#define stbir__1_coeff_only() \
4850 stbir__simdf tot,c,d; \
4851 STBIR_SIMD_NO_UNROLL(decode); \
4852 stbir__simdf_load1z( c, hc ); \
4853 stbir__simdf_0123to0011( c, c ); \
4854 stbir__simdf_load2( d, decode ); \
4855 stbir__simdf_mult( tot, d, c );
4856
4857#define stbir__2_coeff_only() \
4858 stbir__simdf tot,c; \
4859 STBIR_SIMD_NO_UNROLL(decode); \
4860 stbir__simdf_load2( c, hc ); \
4861 stbir__simdf_0123to0011( c, c ); \
4862 stbir__simdf_mult_mem( tot, c, decode );
4863
4864#define stbir__3_coeff_only() \
4865 stbir__simdf tot,c,cs,d; \
4866 STBIR_SIMD_NO_UNROLL(decode); \
4867 stbir__simdf_load( cs, hc ); \
4868 stbir__simdf_0123to0011( c, cs ); \
4869 stbir__simdf_mult_mem( tot, c, decode ); \
4870 stbir__simdf_0123to2222( c, cs ); \
4871 stbir__simdf_load2z( d, decode+4 ); \
4872 stbir__simdf_madd( tot, tot, d, c );
4873
4874#define stbir__store_output_tiny() \
4875 stbir__simdf_0123to2301( c, tot ); \
4876 stbir__simdf_add( tot, tot, c ); \
4877 stbir__simdf_store2( output, tot ); \
4878 horizontal_coefficients += coefficient_width; \
4879 ++horizontal_contributors; \
4880 output += 2;
4881
4882#ifdef STBIR_SIMD8
4883
4884#define stbir__4_coeff_start() \
4885 stbir__simdf8 tot0,c,cs; \
4886 STBIR_SIMD_NO_UNROLL(decode); \
4887 stbir__simdf8_load4b( cs, hc ); \
4888 stbir__simdf8_0123to00112233( c, cs ); \
4889 stbir__simdf8_mult_mem( tot0, c, decode );
4890
4891#define stbir__4_coeff_continue_from_4( ofs ) \
4892 STBIR_SIMD_NO_UNROLL(decode); \
4893 stbir__simdf8_load4b( cs, hc + (ofs) ); \
4894 stbir__simdf8_0123to00112233( c, cs ); \
4895 stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*2 );
4896
4897#define stbir__1_coeff_remnant( ofs ) \
4898 { stbir__simdf t,d; \
4899 stbir__simdf_load1z( t, hc + (ofs) ); \
4900 stbir__simdf_load2( d, decode + (ofs) * 2 ); \
4901 stbir__simdf_0123to0011( t, t ); \
4902 stbir__simdf_mult( t, t, d ); \
4903 stbir__simdf8_add4( tot0, tot0, t ); }
4904
4905#define stbir__2_coeff_remnant( ofs ) \
4906 { stbir__simdf t; \
4907 stbir__simdf_load2( t, hc + (ofs) ); \
4908 stbir__simdf_0123to0011( t, t ); \
4909 stbir__simdf_mult_mem( t, t, decode+(ofs)*2 ); \
4910 stbir__simdf8_add4( tot0, tot0, t ); }
4911
4912#define stbir__3_coeff_remnant( ofs ) \
4913 { stbir__simdf8 d; \
4914 stbir__simdf8_load4b( cs, hc + (ofs) ); \
4915 stbir__simdf8_0123to00112233( c, cs ); \
4916 stbir__simdf8_load6z( d, decode+(ofs)*2 ); \
4917 stbir__simdf8_madd( tot0, tot0, c, d ); }
4918
4919#define stbir__store_output() \
4920 { stbir__simdf t,d; \
4921 stbir__simdf8_add4halves( t, stbir__if_simdf8_cast_to_simdf4(tot0), tot0 ); \
4922 stbir__simdf_0123to2301( d, t ); \
4923 stbir__simdf_add( t, t, d ); \
4924 stbir__simdf_store2( output, t ); \
4925 horizontal_coefficients += coefficient_width; \
4926 ++horizontal_contributors; \
4927 output += 2; }
4928
4929#else
4930
4931#define stbir__4_coeff_start() \
4932 stbir__simdf tot0,tot1,c,cs; \
4933 STBIR_SIMD_NO_UNROLL(decode); \
4934 stbir__simdf_load( cs, hc ); \
4935 stbir__simdf_0123to0011( c, cs ); \
4936 stbir__simdf_mult_mem( tot0, c, decode ); \
4937 stbir__simdf_0123to2233( c, cs ); \
4938 stbir__simdf_mult_mem( tot1, c, decode+4 );
4939
4940#define stbir__4_coeff_continue_from_4( ofs ) \
4941 STBIR_SIMD_NO_UNROLL(decode); \
4942 stbir__simdf_load( cs, hc + (ofs) ); \
4943 stbir__simdf_0123to0011( c, cs ); \
4944 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*2 ); \
4945 stbir__simdf_0123to2233( c, cs ); \
4946 stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*2+4 );
4947
4948#define stbir__1_coeff_remnant( ofs ) \
4949 { stbir__simdf d; \
4950 stbir__simdf_load1z( cs, hc + (ofs) ); \
4951 stbir__simdf_0123to0011( c, cs ); \
4952 stbir__simdf_load2( d, decode + (ofs) * 2 ); \
4953 stbir__simdf_madd( tot0, tot0, d, c ); }
4954
4955#define stbir__2_coeff_remnant( ofs ) \
4956 stbir__simdf_load2( cs, hc + (ofs) ); \
4957 stbir__simdf_0123to0011( c, cs ); \
4958 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*2 );
4959
4960#define stbir__3_coeff_remnant( ofs ) \
4961 { stbir__simdf d; \
4962 stbir__simdf_load( cs, hc + (ofs) ); \
4963 stbir__simdf_0123to0011( c, cs ); \
4964 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*2 ); \
4965 stbir__simdf_0123to2222( c, cs ); \
4966 stbir__simdf_load2z( d, decode + (ofs) * 2 + 4 ); \
4967 stbir__simdf_madd( tot1, tot1, d, c ); }
4968
4969#define stbir__store_output() \
4970 stbir__simdf_add( tot0, tot0, tot1 ); \
4971 stbir__simdf_0123to2301( c, tot0 ); \
4972 stbir__simdf_add( tot0, tot0, c ); \
4973 stbir__simdf_store2( output, tot0 ); \
4974 horizontal_coefficients += coefficient_width; \
4975 ++horizontal_contributors; \
4976 output += 2;
4977
4978#endif
4979
4980#else
4981
4982#define stbir__1_coeff_only() \
4983 float tota,totb,c; \
4984 c = hc[0]; \
4985 tota = decode[0]*c; \
4986 totb = decode[1]*c;
4987
4988#define stbir__2_coeff_only() \
4989 float tota,totb,c; \
4990 c = hc[0]; \
4991 tota = decode[0]*c; \
4992 totb = decode[1]*c; \
4993 c = hc[1]; \
4994 tota += decode[2]*c; \
4995 totb += decode[3]*c;
4996
4997// this weird order of add matches the simd
4998#define stbir__3_coeff_only() \
4999 float tota,totb,c; \
5000 c = hc[0]; \
5001 tota = decode[0]*c; \
5002 totb = decode[1]*c; \
5003 c = hc[2]; \
5004 tota += decode[4]*c; \
5005 totb += decode[5]*c; \
5006 c = hc[1]; \
5007 tota += decode[2]*c; \
5008 totb += decode[3]*c;
5009
5010#define stbir__store_output_tiny() \
5011 output[0] = tota; \
5012 output[1] = totb; \
5013 horizontal_coefficients += coefficient_width; \
5014 ++horizontal_contributors; \
5015 output += 2;
5016
5017#define stbir__4_coeff_start() \
5018 float tota0,tota1,tota2,tota3,totb0,totb1,totb2,totb3,c; \
5019 c = hc[0]; \
5020 tota0 = decode[0]*c; \
5021 totb0 = decode[1]*c; \
5022 c = hc[1]; \
5023 tota1 = decode[2]*c; \
5024 totb1 = decode[3]*c; \
5025 c = hc[2]; \
5026 tota2 = decode[4]*c; \
5027 totb2 = decode[5]*c; \
5028 c = hc[3]; \
5029 tota3 = decode[6]*c; \
5030 totb3 = decode[7]*c;
5031
5032#define stbir__4_coeff_continue_from_4( ofs ) \
5033 c = hc[0+(ofs)]; \
5034 tota0 += decode[0+(ofs)*2]*c; \
5035 totb0 += decode[1+(ofs)*2]*c; \
5036 c = hc[1+(ofs)]; \
5037 tota1 += decode[2+(ofs)*2]*c; \
5038 totb1 += decode[3+(ofs)*2]*c; \
5039 c = hc[2+(ofs)]; \
5040 tota2 += decode[4+(ofs)*2]*c; \
5041 totb2 += decode[5+(ofs)*2]*c; \
5042 c = hc[3+(ofs)]; \
5043 tota3 += decode[6+(ofs)*2]*c; \
5044 totb3 += decode[7+(ofs)*2]*c;
5045
5046#define stbir__1_coeff_remnant( ofs ) \
5047 c = hc[0+(ofs)]; \
5048 tota0 += decode[0+(ofs)*2] * c; \
5049 totb0 += decode[1+(ofs)*2] * c;
5050
5051#define stbir__2_coeff_remnant( ofs ) \
5052 c = hc[0+(ofs)]; \
5053 tota0 += decode[0+(ofs)*2] * c; \
5054 totb0 += decode[1+(ofs)*2] * c; \
5055 c = hc[1+(ofs)]; \
5056 tota1 += decode[2+(ofs)*2] * c; \
5057 totb1 += decode[3+(ofs)*2] * c;
5058
5059#define stbir__3_coeff_remnant( ofs ) \
5060 c = hc[0+(ofs)]; \
5061 tota0 += decode[0+(ofs)*2] * c; \
5062 totb0 += decode[1+(ofs)*2] * c; \
5063 c = hc[1+(ofs)]; \
5064 tota1 += decode[2+(ofs)*2] * c; \
5065 totb1 += decode[3+(ofs)*2] * c; \
5066 c = hc[2+(ofs)]; \
5067 tota2 += decode[4+(ofs)*2] * c; \
5068 totb2 += decode[5+(ofs)*2] * c;
5069
5070#define stbir__store_output() \
5071 output[0] = (tota0+tota2)+(tota1+tota3); \
5072 output[1] = (totb0+totb2)+(totb1+totb3); \
5073 horizontal_coefficients += coefficient_width; \
5074 ++horizontal_contributors; \
5075 output += 2;
5076
5077#endif
5078
5079#define STBIR__horizontal_channels 2
5080#define STB_IMAGE_RESIZE_DO_HORIZONTALS
5081#include STBIR__HEADER_FILENAME
5082
5083
5084//=================
5085// Do 3 channel horizontal routines
5086
5087#ifdef STBIR_SIMD
5088
5089#define stbir__1_coeff_only() \
5090 stbir__simdf tot,c,d; \
5091 STBIR_SIMD_NO_UNROLL(decode); \
5092 stbir__simdf_load1z( c, hc ); \
5093 stbir__simdf_0123to0001( c, c ); \
5094 stbir__simdf_load( d, decode ); \
5095 stbir__simdf_mult( tot, d, c );
5096
5097#define stbir__2_coeff_only() \
5098 stbir__simdf tot,c,cs,d; \
5099 STBIR_SIMD_NO_UNROLL(decode); \
5100 stbir__simdf_load2( cs, hc ); \
5101 stbir__simdf_0123to0000( c, cs ); \
5102 stbir__simdf_load( d, decode ); \
5103 stbir__simdf_mult( tot, d, c ); \
5104 stbir__simdf_0123to1111( c, cs ); \
5105 stbir__simdf_load( d, decode+3 ); \
5106 stbir__simdf_madd( tot, tot, d, c );
5107
5108#define stbir__3_coeff_only() \
5109 stbir__simdf tot,c,d,cs; \
5110 STBIR_SIMD_NO_UNROLL(decode); \
5111 stbir__simdf_load( cs, hc ); \
5112 stbir__simdf_0123to0000( c, cs ); \
5113 stbir__simdf_load( d, decode ); \
5114 stbir__simdf_mult( tot, d, c ); \
5115 stbir__simdf_0123to1111( c, cs ); \
5116 stbir__simdf_load( d, decode+3 ); \
5117 stbir__simdf_madd( tot, tot, d, c ); \
5118 stbir__simdf_0123to2222( c, cs ); \
5119 stbir__simdf_load( d, decode+6 ); \
5120 stbir__simdf_madd( tot, tot, d, c );
5121
5122#define stbir__store_output_tiny() \
5123 stbir__simdf_store2( output, tot ); \
5124 stbir__simdf_0123to2301( tot, tot ); \
5125 stbir__simdf_store1( output+2, tot ); \
5126 horizontal_coefficients += coefficient_width; \
5127 ++horizontal_contributors; \
5128 output += 3;
5129
5130#ifdef STBIR_SIMD8
5131
5132// we're loading from the XXXYYY decode by -1 to get the XXXYYY into different halves of the AVX reg fyi
5133#define stbir__4_coeff_start() \
5134 stbir__simdf8 tot0,tot1,c,cs; stbir__simdf t; \
5135 STBIR_SIMD_NO_UNROLL(decode); \
5136 stbir__simdf8_load4b( cs, hc ); \
5137 stbir__simdf8_0123to00001111( c, cs ); \
5138 stbir__simdf8_mult_mem( tot0, c, decode - 1 ); \
5139 stbir__simdf8_0123to22223333( c, cs ); \
5140 stbir__simdf8_mult_mem( tot1, c, decode+6 - 1 );
5141
5142#define stbir__4_coeff_continue_from_4( ofs ) \
5143 STBIR_SIMD_NO_UNROLL(decode); \
5144 stbir__simdf8_load4b( cs, hc + (ofs) ); \
5145 stbir__simdf8_0123to00001111( c, cs ); \
5146 stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*3 - 1 ); \
5147 stbir__simdf8_0123to22223333( c, cs ); \
5148 stbir__simdf8_madd_mem( tot1, tot1, c, decode+(ofs)*3 + 6 - 1 );
5149
5150#define stbir__1_coeff_remnant( ofs ) \
5151 STBIR_SIMD_NO_UNROLL(decode); \
5152 stbir__simdf_load1rep4( t, hc + (ofs) ); \
5153 stbir__simdf8_madd_mem4( tot0, tot0, t, decode+(ofs)*3 - 1 );
5154
5155#define stbir__2_coeff_remnant( ofs ) \
5156 STBIR_SIMD_NO_UNROLL(decode); \
5157 stbir__simdf8_load4b( cs, hc + (ofs) - 2 ); \
5158 stbir__simdf8_0123to22223333( c, cs ); \
5159 stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*3 - 1 );
5160
5161 #define stbir__3_coeff_remnant( ofs ) \
5162 STBIR_SIMD_NO_UNROLL(decode); \
5163 stbir__simdf8_load4b( cs, hc + (ofs) ); \
5164 stbir__simdf8_0123to00001111( c, cs ); \
5165 stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*3 - 1 ); \
5166 stbir__simdf8_0123to2222( t, cs ); \
5167 stbir__simdf8_madd_mem4( tot1, tot1, t, decode+(ofs)*3 + 6 - 1 );
5168
5169#define stbir__store_output() \
5170 stbir__simdf8_add( tot0, tot0, tot1 ); \
5171 stbir__simdf_0123to1230( t, stbir__if_simdf8_cast_to_simdf4( tot0 ) ); \
5172 stbir__simdf8_add4halves( t, t, tot0 ); \
5173 horizontal_coefficients += coefficient_width; \
5174 ++horizontal_contributors; \
5175 output += 3; \
5176 if ( output < output_end ) \
5177 { \
5178 stbir__simdf_store( output-3, t ); \
5179 continue; \
5180 } \
5181 { stbir__simdf tt; stbir__simdf_0123to2301( tt, t ); \
5182 stbir__simdf_store2( output-3, t ); \
5183 stbir__simdf_store1( output+2-3, tt ); } \
5184 break;
5185
5186
5187#else
5188
5189#define stbir__4_coeff_start() \
5190 stbir__simdf tot0,tot1,tot2,c,cs; \
5191 STBIR_SIMD_NO_UNROLL(decode); \
5192 stbir__simdf_load( cs, hc ); \
5193 stbir__simdf_0123to0001( c, cs ); \
5194 stbir__simdf_mult_mem( tot0, c, decode ); \
5195 stbir__simdf_0123to1122( c, cs ); \
5196 stbir__simdf_mult_mem( tot1, c, decode+4 ); \
5197 stbir__simdf_0123to2333( c, cs ); \
5198 stbir__simdf_mult_mem( tot2, c, decode+8 );
5199
5200#define stbir__4_coeff_continue_from_4( ofs ) \
5201 STBIR_SIMD_NO_UNROLL(decode); \
5202 stbir__simdf_load( cs, hc + (ofs) ); \
5203 stbir__simdf_0123to0001( c, cs ); \
5204 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*3 ); \
5205 stbir__simdf_0123to1122( c, cs ); \
5206 stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*3+4 ); \
5207 stbir__simdf_0123to2333( c, cs ); \
5208 stbir__simdf_madd_mem( tot2, tot2, c, decode+(ofs)*3+8 );
5209
5210#define stbir__1_coeff_remnant( ofs ) \
5211 STBIR_SIMD_NO_UNROLL(decode); \
5212 stbir__simdf_load1z( c, hc + (ofs) ); \
5213 stbir__simdf_0123to0001( c, c ); \
5214 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*3 );
5215
5216#define stbir__2_coeff_remnant( ofs ) \
5217 { stbir__simdf d; \
5218 STBIR_SIMD_NO_UNROLL(decode); \
5219 stbir__simdf_load2z( cs, hc + (ofs) ); \
5220 stbir__simdf_0123to0001( c, cs ); \
5221 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*3 ); \
5222 stbir__simdf_0123to1122( c, cs ); \
5223 stbir__simdf_load2z( d, decode+(ofs)*3+4 ); \
5224 stbir__simdf_madd( tot1, tot1, c, d ); }
5225
5226#define stbir__3_coeff_remnant( ofs ) \
5227 { stbir__simdf d; \
5228 STBIR_SIMD_NO_UNROLL(decode); \
5229 stbir__simdf_load( cs, hc + (ofs) ); \
5230 stbir__simdf_0123to0001( c, cs ); \
5231 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*3 ); \
5232 stbir__simdf_0123to1122( c, cs ); \
5233 stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*3+4 ); \
5234 stbir__simdf_0123to2222( c, cs ); \
5235 stbir__simdf_load1z( d, decode+(ofs)*3+8 ); \
5236 stbir__simdf_madd( tot2, tot2, c, d ); }
5237
5238#define stbir__store_output() \
5239 stbir__simdf_0123ABCDto3ABx( c, tot0, tot1 ); \
5240 stbir__simdf_0123ABCDto23Ax( cs, tot1, tot2 ); \
5241 stbir__simdf_0123to1230( tot2, tot2 ); \
5242 stbir__simdf_add( tot0, tot0, cs ); \
5243 stbir__simdf_add( c, c, tot2 ); \
5244 stbir__simdf_add( tot0, tot0, c ); \
5245 horizontal_coefficients += coefficient_width; \
5246 ++horizontal_contributors; \
5247 output += 3; \
5248 if ( output < output_end ) \
5249 { \
5250 stbir__simdf_store( output-3, tot0 ); \
5251 continue; \
5252 } \
5253 stbir__simdf_0123to2301( tot1, tot0 ); \
5254 stbir__simdf_store2( output-3, tot0 ); \
5255 stbir__simdf_store1( output+2-3, tot1 ); \
5256 break;
5257
5258#endif
5259
5260#else
5261
5262#define stbir__1_coeff_only() \
5263 float tot0, tot1, tot2, c; \
5264 c = hc[0]; \
5265 tot0 = decode[0]*c; \
5266 tot1 = decode[1]*c; \
5267 tot2 = decode[2]*c;
5268
5269#define stbir__2_coeff_only() \
5270 float tot0, tot1, tot2, c; \
5271 c = hc[0]; \
5272 tot0 = decode[0]*c; \
5273 tot1 = decode[1]*c; \
5274 tot2 = decode[2]*c; \
5275 c = hc[1]; \
5276 tot0 += decode[3]*c; \
5277 tot1 += decode[4]*c; \
5278 tot2 += decode[5]*c;
5279
5280#define stbir__3_coeff_only() \
5281 float tot0, tot1, tot2, c; \
5282 c = hc[0]; \
5283 tot0 = decode[0]*c; \
5284 tot1 = decode[1]*c; \
5285 tot2 = decode[2]*c; \
5286 c = hc[1]; \
5287 tot0 += decode[3]*c; \
5288 tot1 += decode[4]*c; \
5289 tot2 += decode[5]*c; \
5290 c = hc[2]; \
5291 tot0 += decode[6]*c; \
5292 tot1 += decode[7]*c; \
5293 tot2 += decode[8]*c;
5294
5295#define stbir__store_output_tiny() \
5296 output[0] = tot0; \
5297 output[1] = tot1; \
5298 output[2] = tot2; \
5299 horizontal_coefficients += coefficient_width; \
5300 ++horizontal_contributors; \
5301 output += 3;
5302
5303#define stbir__4_coeff_start() \
5304 float tota0,tota1,tota2,totb0,totb1,totb2,totc0,totc1,totc2,totd0,totd1,totd2,c; \
5305 c = hc[0]; \
5306 tota0 = decode[0]*c; \
5307 tota1 = decode[1]*c; \
5308 tota2 = decode[2]*c; \
5309 c = hc[1]; \
5310 totb0 = decode[3]*c; \
5311 totb1 = decode[4]*c; \
5312 totb2 = decode[5]*c; \
5313 c = hc[2]; \
5314 totc0 = decode[6]*c; \
5315 totc1 = decode[7]*c; \
5316 totc2 = decode[8]*c; \
5317 c = hc[3]; \
5318 totd0 = decode[9]*c; \
5319 totd1 = decode[10]*c; \
5320 totd2 = decode[11]*c;
5321
5322#define stbir__4_coeff_continue_from_4( ofs ) \
5323 c = hc[0+(ofs)]; \
5324 tota0 += decode[0+(ofs)*3]*c; \
5325 tota1 += decode[1+(ofs)*3]*c; \
5326 tota2 += decode[2+(ofs)*3]*c; \
5327 c = hc[1+(ofs)]; \
5328 totb0 += decode[3+(ofs)*3]*c; \
5329 totb1 += decode[4+(ofs)*3]*c; \
5330 totb2 += decode[5+(ofs)*3]*c; \
5331 c = hc[2+(ofs)]; \
5332 totc0 += decode[6+(ofs)*3]*c; \
5333 totc1 += decode[7+(ofs)*3]*c; \
5334 totc2 += decode[8+(ofs)*3]*c; \
5335 c = hc[3+(ofs)]; \
5336 totd0 += decode[9+(ofs)*3]*c; \
5337 totd1 += decode[10+(ofs)*3]*c; \
5338 totd2 += decode[11+(ofs)*3]*c;
5339
5340#define stbir__1_coeff_remnant( ofs ) \
5341 c = hc[0+(ofs)]; \
5342 tota0 += decode[0+(ofs)*3]*c; \
5343 tota1 += decode[1+(ofs)*3]*c; \
5344 tota2 += decode[2+(ofs)*3]*c;
5345
5346#define stbir__2_coeff_remnant( ofs ) \
5347 c = hc[0+(ofs)]; \
5348 tota0 += decode[0+(ofs)*3]*c; \
5349 tota1 += decode[1+(ofs)*3]*c; \
5350 tota2 += decode[2+(ofs)*3]*c; \
5351 c = hc[1+(ofs)]; \
5352 totb0 += decode[3+(ofs)*3]*c; \
5353 totb1 += decode[4+(ofs)*3]*c; \
5354 totb2 += decode[5+(ofs)*3]*c; \
5355
5356#define stbir__3_coeff_remnant( ofs ) \
5357 c = hc[0+(ofs)]; \
5358 tota0 += decode[0+(ofs)*3]*c; \
5359 tota1 += decode[1+(ofs)*3]*c; \
5360 tota2 += decode[2+(ofs)*3]*c; \
5361 c = hc[1+(ofs)]; \
5362 totb0 += decode[3+(ofs)*3]*c; \
5363 totb1 += decode[4+(ofs)*3]*c; \
5364 totb2 += decode[5+(ofs)*3]*c; \
5365 c = hc[2+(ofs)]; \
5366 totc0 += decode[6+(ofs)*3]*c; \
5367 totc1 += decode[7+(ofs)*3]*c; \
5368 totc2 += decode[8+(ofs)*3]*c;
5369
5370#define stbir__store_output() \
5371 output[0] = (tota0+totc0)+(totb0+totd0); \
5372 output[1] = (tota1+totc1)+(totb1+totd1); \
5373 output[2] = (tota2+totc2)+(totb2+totd2); \
5374 horizontal_coefficients += coefficient_width; \
5375 ++horizontal_contributors; \
5376 output += 3;
5377
5378#endif
5379
5380#define STBIR__horizontal_channels 3
5381#define STB_IMAGE_RESIZE_DO_HORIZONTALS
5382#include STBIR__HEADER_FILENAME
5383
5384//=================
5385// Do 4 channel horizontal routines
5386
5387#ifdef STBIR_SIMD
5388
5389#define stbir__1_coeff_only() \
5390 stbir__simdf tot,c; \
5391 STBIR_SIMD_NO_UNROLL(decode); \
5392 stbir__simdf_load1( c, hc ); \
5393 stbir__simdf_0123to0000( c, c ); \
5394 stbir__simdf_mult_mem( tot, c, decode );
5395
5396#define stbir__2_coeff_only() \
5397 stbir__simdf tot,c,cs; \
5398 STBIR_SIMD_NO_UNROLL(decode); \
5399 stbir__simdf_load2( cs, hc ); \
5400 stbir__simdf_0123to0000( c, cs ); \
5401 stbir__simdf_mult_mem( tot, c, decode ); \
5402 stbir__simdf_0123to1111( c, cs ); \
5403 stbir__simdf_madd_mem( tot, tot, c, decode+4 );
5404
5405#define stbir__3_coeff_only() \
5406 stbir__simdf tot,c,cs; \
5407 STBIR_SIMD_NO_UNROLL(decode); \
5408 stbir__simdf_load( cs, hc ); \
5409 stbir__simdf_0123to0000( c, cs ); \
5410 stbir__simdf_mult_mem( tot, c, decode ); \
5411 stbir__simdf_0123to1111( c, cs ); \
5412 stbir__simdf_madd_mem( tot, tot, c, decode+4 ); \
5413 stbir__simdf_0123to2222( c, cs ); \
5414 stbir__simdf_madd_mem( tot, tot, c, decode+8 );
5415
5416#define stbir__store_output_tiny() \
5417 stbir__simdf_store( output, tot ); \
5418 horizontal_coefficients += coefficient_width; \
5419 ++horizontal_contributors; \
5420 output += 4;
5421
5422#ifdef STBIR_SIMD8
5423
5424#define stbir__4_coeff_start() \
5425 stbir__simdf8 tot0,c,cs; stbir__simdf t; \
5426 STBIR_SIMD_NO_UNROLL(decode); \
5427 stbir__simdf8_load4b( cs, hc ); \
5428 stbir__simdf8_0123to00001111( c, cs ); \
5429 stbir__simdf8_mult_mem( tot0, c, decode ); \
5430 stbir__simdf8_0123to22223333( c, cs ); \
5431 stbir__simdf8_madd_mem( tot0, tot0, c, decode+8 );
5432
5433#define stbir__4_coeff_continue_from_4( ofs ) \
5434 STBIR_SIMD_NO_UNROLL(decode); \
5435 stbir__simdf8_load4b( cs, hc + (ofs) ); \
5436 stbir__simdf8_0123to00001111( c, cs ); \
5437 stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*4 ); \
5438 stbir__simdf8_0123to22223333( c, cs ); \
5439 stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*4+8 );
5440
5441#define stbir__1_coeff_remnant( ofs ) \
5442 STBIR_SIMD_NO_UNROLL(decode); \
5443 stbir__simdf_load1rep4( t, hc + (ofs) ); \
5444 stbir__simdf8_madd_mem4( tot0, tot0, t, decode+(ofs)*4 );
5445
5446#define stbir__2_coeff_remnant( ofs ) \
5447 STBIR_SIMD_NO_UNROLL(decode); \
5448 stbir__simdf8_load4b( cs, hc + (ofs) - 2 ); \
5449 stbir__simdf8_0123to22223333( c, cs ); \
5450 stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*4 );
5451
5452 #define stbir__3_coeff_remnant( ofs ) \
5453 STBIR_SIMD_NO_UNROLL(decode); \
5454 stbir__simdf8_load4b( cs, hc + (ofs) ); \
5455 stbir__simdf8_0123to00001111( c, cs ); \
5456 stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*4 ); \
5457 stbir__simdf8_0123to2222( t, cs ); \
5458 stbir__simdf8_madd_mem4( tot0, tot0, t, decode+(ofs)*4+8 );
5459
5460#define stbir__store_output() \
5461 stbir__simdf8_add4halves( t, stbir__if_simdf8_cast_to_simdf4(tot0), tot0 ); \
5462 stbir__simdf_store( output, t ); \
5463 horizontal_coefficients += coefficient_width; \
5464 ++horizontal_contributors; \
5465 output += 4;
5466
5467#else
5468
5469#define stbir__4_coeff_start() \
5470 stbir__simdf tot0,tot1,c,cs; \
5471 STBIR_SIMD_NO_UNROLL(decode); \
5472 stbir__simdf_load( cs, hc ); \
5473 stbir__simdf_0123to0000( c, cs ); \
5474 stbir__simdf_mult_mem( tot0, c, decode ); \
5475 stbir__simdf_0123to1111( c, cs ); \
5476 stbir__simdf_mult_mem( tot1, c, decode+4 ); \
5477 stbir__simdf_0123to2222( c, cs ); \
5478 stbir__simdf_madd_mem( tot0, tot0, c, decode+8 ); \
5479 stbir__simdf_0123to3333( c, cs ); \
5480 stbir__simdf_madd_mem( tot1, tot1, c, decode+12 );
5481
5482#define stbir__4_coeff_continue_from_4( ofs ) \
5483 STBIR_SIMD_NO_UNROLL(decode); \
5484 stbir__simdf_load( cs, hc + (ofs) ); \
5485 stbir__simdf_0123to0000( c, cs ); \
5486 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*4 ); \
5487 stbir__simdf_0123to1111( c, cs ); \
5488 stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*4+4 ); \
5489 stbir__simdf_0123to2222( c, cs ); \
5490 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*4+8 ); \
5491 stbir__simdf_0123to3333( c, cs ); \
5492 stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*4+12 );
5493
5494#define stbir__1_coeff_remnant( ofs ) \
5495 STBIR_SIMD_NO_UNROLL(decode); \
5496 stbir__simdf_load1( c, hc + (ofs) ); \
5497 stbir__simdf_0123to0000( c, c ); \
5498 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*4 );
5499
5500#define stbir__2_coeff_remnant( ofs ) \
5501 STBIR_SIMD_NO_UNROLL(decode); \
5502 stbir__simdf_load2( cs, hc + (ofs) ); \
5503 stbir__simdf_0123to0000( c, cs ); \
5504 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*4 ); \
5505 stbir__simdf_0123to1111( c, cs ); \
5506 stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*4+4 );
5507
5508#define stbir__3_coeff_remnant( ofs ) \
5509 STBIR_SIMD_NO_UNROLL(decode); \
5510 stbir__simdf_load( cs, hc + (ofs) ); \
5511 stbir__simdf_0123to0000( c, cs ); \
5512 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*4 ); \
5513 stbir__simdf_0123to1111( c, cs ); \
5514 stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*4+4 ); \
5515 stbir__simdf_0123to2222( c, cs ); \
5516 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*4+8 );
5517
5518#define stbir__store_output() \
5519 stbir__simdf_add( tot0, tot0, tot1 ); \
5520 stbir__simdf_store( output, tot0 ); \
5521 horizontal_coefficients += coefficient_width; \
5522 ++horizontal_contributors; \
5523 output += 4;
5524
5525#endif
5526
5527#else
5528
5529#define stbir__1_coeff_only() \
5530 float p0,p1,p2,p3,c; \
5531 STBIR_SIMD_NO_UNROLL(decode); \
5532 c = hc[0]; \
5533 p0 = decode[0] * c; \
5534 p1 = decode[1] * c; \
5535 p2 = decode[2] * c; \
5536 p3 = decode[3] * c;
5537
5538#define stbir__2_coeff_only() \
5539 float p0,p1,p2,p3,c; \
5540 STBIR_SIMD_NO_UNROLL(decode); \
5541 c = hc[0]; \
5542 p0 = decode[0] * c; \
5543 p1 = decode[1] * c; \
5544 p2 = decode[2] * c; \
5545 p3 = decode[3] * c; \
5546 c = hc[1]; \
5547 p0 += decode[4] * c; \
5548 p1 += decode[5] * c; \
5549 p2 += decode[6] * c; \
5550 p3 += decode[7] * c;
5551
5552#define stbir__3_coeff_only() \
5553 float p0,p1,p2,p3,c; \
5554 STBIR_SIMD_NO_UNROLL(decode); \
5555 c = hc[0]; \
5556 p0 = decode[0] * c; \
5557 p1 = decode[1] * c; \
5558 p2 = decode[2] * c; \
5559 p3 = decode[3] * c; \
5560 c = hc[1]; \
5561 p0 += decode[4] * c; \
5562 p1 += decode[5] * c; \
5563 p2 += decode[6] * c; \
5564 p3 += decode[7] * c; \
5565 c = hc[2]; \
5566 p0 += decode[8] * c; \
5567 p1 += decode[9] * c; \
5568 p2 += decode[10] * c; \
5569 p3 += decode[11] * c;
5570
5571#define stbir__store_output_tiny() \
5572 output[0] = p0; \
5573 output[1] = p1; \
5574 output[2] = p2; \
5575 output[3] = p3; \
5576 horizontal_coefficients += coefficient_width; \
5577 ++horizontal_contributors; \
5578 output += 4;
5579
5580#define stbir__4_coeff_start() \
5581 float x0,x1,x2,x3,y0,y1,y2,y3,c; \
5582 STBIR_SIMD_NO_UNROLL(decode); \
5583 c = hc[0]; \
5584 x0 = decode[0] * c; \
5585 x1 = decode[1] * c; \
5586 x2 = decode[2] * c; \
5587 x3 = decode[3] * c; \
5588 c = hc[1]; \
5589 y0 = decode[4] * c; \
5590 y1 = decode[5] * c; \
5591 y2 = decode[6] * c; \
5592 y3 = decode[7] * c; \
5593 c = hc[2]; \
5594 x0 += decode[8] * c; \
5595 x1 += decode[9] * c; \
5596 x2 += decode[10] * c; \
5597 x3 += decode[11] * c; \
5598 c = hc[3]; \
5599 y0 += decode[12] * c; \
5600 y1 += decode[13] * c; \
5601 y2 += decode[14] * c; \
5602 y3 += decode[15] * c;
5603
5604#define stbir__4_coeff_continue_from_4( ofs ) \
5605 STBIR_SIMD_NO_UNROLL(decode); \
5606 c = hc[0+(ofs)]; \
5607 x0 += decode[0+(ofs)*4] * c; \
5608 x1 += decode[1+(ofs)*4] * c; \
5609 x2 += decode[2+(ofs)*4] * c; \
5610 x3 += decode[3+(ofs)*4] * c; \
5611 c = hc[1+(ofs)]; \
5612 y0 += decode[4+(ofs)*4] * c; \
5613 y1 += decode[5+(ofs)*4] * c; \
5614 y2 += decode[6+(ofs)*4] * c; \
5615 y3 += decode[7+(ofs)*4] * c; \
5616 c = hc[2+(ofs)]; \
5617 x0 += decode[8+(ofs)*4] * c; \
5618 x1 += decode[9+(ofs)*4] * c; \
5619 x2 += decode[10+(ofs)*4] * c; \
5620 x3 += decode[11+(ofs)*4] * c; \
5621 c = hc[3+(ofs)]; \
5622 y0 += decode[12+(ofs)*4] * c; \
5623 y1 += decode[13+(ofs)*4] * c; \
5624 y2 += decode[14+(ofs)*4] * c; \
5625 y3 += decode[15+(ofs)*4] * c;
5626
5627#define stbir__1_coeff_remnant( ofs ) \
5628 STBIR_SIMD_NO_UNROLL(decode); \
5629 c = hc[0+(ofs)]; \
5630 x0 += decode[0+(ofs)*4] * c; \
5631 x1 += decode[1+(ofs)*4] * c; \
5632 x2 += decode[2+(ofs)*4] * c; \
5633 x3 += decode[3+(ofs)*4] * c;
5634
5635#define stbir__2_coeff_remnant( ofs ) \
5636 STBIR_SIMD_NO_UNROLL(decode); \
5637 c = hc[0+(ofs)]; \
5638 x0 += decode[0+(ofs)*4] * c; \
5639 x1 += decode[1+(ofs)*4] * c; \
5640 x2 += decode[2+(ofs)*4] * c; \
5641 x3 += decode[3+(ofs)*4] * c; \
5642 c = hc[1+(ofs)]; \
5643 y0 += decode[4+(ofs)*4] * c; \
5644 y1 += decode[5+(ofs)*4] * c; \
5645 y2 += decode[6+(ofs)*4] * c; \
5646 y3 += decode[7+(ofs)*4] * c;
5647
5648#define stbir__3_coeff_remnant( ofs ) \
5649 STBIR_SIMD_NO_UNROLL(decode); \
5650 c = hc[0+(ofs)]; \
5651 x0 += decode[0+(ofs)*4] * c; \
5652 x1 += decode[1+(ofs)*4] * c; \
5653 x2 += decode[2+(ofs)*4] * c; \
5654 x3 += decode[3+(ofs)*4] * c; \
5655 c = hc[1+(ofs)]; \
5656 y0 += decode[4+(ofs)*4] * c; \
5657 y1 += decode[5+(ofs)*4] * c; \
5658 y2 += decode[6+(ofs)*4] * c; \
5659 y3 += decode[7+(ofs)*4] * c; \
5660 c = hc[2+(ofs)]; \
5661 x0 += decode[8+(ofs)*4] * c; \
5662 x1 += decode[9+(ofs)*4] * c; \
5663 x2 += decode[10+(ofs)*4] * c; \
5664 x3 += decode[11+(ofs)*4] * c;
5665
5666#define stbir__store_output() \
5667 output[0] = x0 + y0; \
5668 output[1] = x1 + y1; \
5669 output[2] = x2 + y2; \
5670 output[3] = x3 + y3; \
5671 horizontal_coefficients += coefficient_width; \
5672 ++horizontal_contributors; \
5673 output += 4;
5674
5675#endif
5676
5677#define STBIR__horizontal_channels 4
5678#define STB_IMAGE_RESIZE_DO_HORIZONTALS
5679#include STBIR__HEADER_FILENAME
5680
5681
5682
5683//=================
5684// Do 7 channel horizontal routines
5685
5686#ifdef STBIR_SIMD
5687
5688#define stbir__1_coeff_only() \
5689 stbir__simdf tot0,tot1,c; \
5690 STBIR_SIMD_NO_UNROLL(decode); \
5691 stbir__simdf_load1( c, hc ); \
5692 stbir__simdf_0123to0000( c, c ); \
5693 stbir__simdf_mult_mem( tot0, c, decode ); \
5694 stbir__simdf_mult_mem( tot1, c, decode+3 );
5695
5696#define stbir__2_coeff_only() \
5697 stbir__simdf tot0,tot1,c,cs; \
5698 STBIR_SIMD_NO_UNROLL(decode); \
5699 stbir__simdf_load2( cs, hc ); \
5700 stbir__simdf_0123to0000( c, cs ); \
5701 stbir__simdf_mult_mem( tot0, c, decode ); \
5702 stbir__simdf_mult_mem( tot1, c, decode+3 ); \
5703 stbir__simdf_0123to1111( c, cs ); \
5704 stbir__simdf_madd_mem( tot0, tot0, c, decode+7 ); \
5705 stbir__simdf_madd_mem( tot1, tot1, c,decode+10 );
5706
5707#define stbir__3_coeff_only() \
5708 stbir__simdf tot0,tot1,c,cs; \
5709 STBIR_SIMD_NO_UNROLL(decode); \
5710 stbir__simdf_load( cs, hc ); \
5711 stbir__simdf_0123to0000( c, cs ); \
5712 stbir__simdf_mult_mem( tot0, c, decode ); \
5713 stbir__simdf_mult_mem( tot1, c, decode+3 ); \
5714 stbir__simdf_0123to1111( c, cs ); \
5715 stbir__simdf_madd_mem( tot0, tot0, c, decode+7 ); \
5716 stbir__simdf_madd_mem( tot1, tot1, c, decode+10 ); \
5717 stbir__simdf_0123to2222( c, cs ); \
5718 stbir__simdf_madd_mem( tot0, tot0, c, decode+14 ); \
5719 stbir__simdf_madd_mem( tot1, tot1, c, decode+17 );
5720
5721#define stbir__store_output_tiny() \
5722 stbir__simdf_store( output+3, tot1 ); \
5723 stbir__simdf_store( output, tot0 ); \
5724 horizontal_coefficients += coefficient_width; \
5725 ++horizontal_contributors; \
5726 output += 7;
5727
5728#ifdef STBIR_SIMD8
5729
5730#define stbir__4_coeff_start() \
5731 stbir__simdf8 tot0,tot1,c,cs; \
5732 STBIR_SIMD_NO_UNROLL(decode); \
5733 stbir__simdf8_load4b( cs, hc ); \
5734 stbir__simdf8_0123to00000000( c, cs ); \
5735 stbir__simdf8_mult_mem( tot0, c, decode ); \
5736 stbir__simdf8_0123to11111111( c, cs ); \
5737 stbir__simdf8_mult_mem( tot1, c, decode+7 ); \
5738 stbir__simdf8_0123to22222222( c, cs ); \
5739 stbir__simdf8_madd_mem( tot0, tot0, c, decode+14 ); \
5740 stbir__simdf8_0123to33333333( c, cs ); \
5741 stbir__simdf8_madd_mem( tot1, tot1, c, decode+21 );
5742
5743#define stbir__4_coeff_continue_from_4( ofs ) \
5744 STBIR_SIMD_NO_UNROLL(decode); \
5745 stbir__simdf8_load4b( cs, hc + (ofs) ); \
5746 stbir__simdf8_0123to00000000( c, cs ); \
5747 stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*7 ); \
5748 stbir__simdf8_0123to11111111( c, cs ); \
5749 stbir__simdf8_madd_mem( tot1, tot1, c, decode+(ofs)*7+7 ); \
5750 stbir__simdf8_0123to22222222( c, cs ); \
5751 stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*7+14 ); \
5752 stbir__simdf8_0123to33333333( c, cs ); \
5753 stbir__simdf8_madd_mem( tot1, tot1, c, decode+(ofs)*7+21 );
5754
5755#define stbir__1_coeff_remnant( ofs ) \
5756 STBIR_SIMD_NO_UNROLL(decode); \
5757 stbir__simdf8_load1b( c, hc + (ofs) ); \
5758 stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*7 );
5759
5760#define stbir__2_coeff_remnant( ofs ) \
5761 STBIR_SIMD_NO_UNROLL(decode); \
5762 stbir__simdf8_load1b( c, hc + (ofs) ); \
5763 stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*7 ); \
5764 stbir__simdf8_load1b( c, hc + (ofs)+1 ); \
5765 stbir__simdf8_madd_mem( tot1, tot1, c, decode+(ofs)*7+7 );
5766
5767#define stbir__3_coeff_remnant( ofs ) \
5768 STBIR_SIMD_NO_UNROLL(decode); \
5769 stbir__simdf8_load4b( cs, hc + (ofs) ); \
5770 stbir__simdf8_0123to00000000( c, cs ); \
5771 stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*7 ); \
5772 stbir__simdf8_0123to11111111( c, cs ); \
5773 stbir__simdf8_madd_mem( tot1, tot1, c, decode+(ofs)*7+7 ); \
5774 stbir__simdf8_0123to22222222( c, cs ); \
5775 stbir__simdf8_madd_mem( tot0, tot0, c, decode+(ofs)*7+14 );
5776
5777#define stbir__store_output() \
5778 stbir__simdf8_add( tot0, tot0, tot1 ); \
5779 horizontal_coefficients += coefficient_width; \
5780 ++horizontal_contributors; \
5781 output += 7; \
5782 if ( output < output_end ) \
5783 { \
5784 stbir__simdf8_store( output-7, tot0 ); \
5785 continue; \
5786 } \
5787 stbir__simdf_store( output-7+3, stbir__simdf_swiz(stbir__simdf8_gettop4(tot0),0,0,1,2) ); \
5788 stbir__simdf_store( output-7, stbir__if_simdf8_cast_to_simdf4(tot0) ); \
5789 break;
5790
5791#else
5792
5793#define stbir__4_coeff_start() \
5794 stbir__simdf tot0,tot1,tot2,tot3,c,cs; \
5795 STBIR_SIMD_NO_UNROLL(decode); \
5796 stbir__simdf_load( cs, hc ); \
5797 stbir__simdf_0123to0000( c, cs ); \
5798 stbir__simdf_mult_mem( tot0, c, decode ); \
5799 stbir__simdf_mult_mem( tot1, c, decode+3 ); \
5800 stbir__simdf_0123to1111( c, cs ); \
5801 stbir__simdf_mult_mem( tot2, c, decode+7 ); \
5802 stbir__simdf_mult_mem( tot3, c, decode+10 ); \
5803 stbir__simdf_0123to2222( c, cs ); \
5804 stbir__simdf_madd_mem( tot0, tot0, c, decode+14 ); \
5805 stbir__simdf_madd_mem( tot1, tot1, c, decode+17 ); \
5806 stbir__simdf_0123to3333( c, cs ); \
5807 stbir__simdf_madd_mem( tot2, tot2, c, decode+21 ); \
5808 stbir__simdf_madd_mem( tot3, tot3, c, decode+24 );
5809
5810#define stbir__4_coeff_continue_from_4( ofs ) \
5811 STBIR_SIMD_NO_UNROLL(decode); \
5812 stbir__simdf_load( cs, hc + (ofs) ); \
5813 stbir__simdf_0123to0000( c, cs ); \
5814 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*7 ); \
5815 stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*7+3 ); \
5816 stbir__simdf_0123to1111( c, cs ); \
5817 stbir__simdf_madd_mem( tot2, tot2, c, decode+(ofs)*7+7 ); \
5818 stbir__simdf_madd_mem( tot3, tot3, c, decode+(ofs)*7+10 ); \
5819 stbir__simdf_0123to2222( c, cs ); \
5820 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*7+14 ); \
5821 stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*7+17 ); \
5822 stbir__simdf_0123to3333( c, cs ); \
5823 stbir__simdf_madd_mem( tot2, tot2, c, decode+(ofs)*7+21 ); \
5824 stbir__simdf_madd_mem( tot3, tot3, c, decode+(ofs)*7+24 );
5825
5826#define stbir__1_coeff_remnant( ofs ) \
5827 STBIR_SIMD_NO_UNROLL(decode); \
5828 stbir__simdf_load1( c, hc + (ofs) ); \
5829 stbir__simdf_0123to0000( c, c ); \
5830 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*7 ); \
5831 stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*7+3 ); \
5832
5833#define stbir__2_coeff_remnant( ofs ) \
5834 STBIR_SIMD_NO_UNROLL(decode); \
5835 stbir__simdf_load2( cs, hc + (ofs) ); \
5836 stbir__simdf_0123to0000( c, cs ); \
5837 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*7 ); \
5838 stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*7+3 ); \
5839 stbir__simdf_0123to1111( c, cs ); \
5840 stbir__simdf_madd_mem( tot2, tot2, c, decode+(ofs)*7+7 ); \
5841 stbir__simdf_madd_mem( tot3, tot3, c, decode+(ofs)*7+10 );
5842
5843#define stbir__3_coeff_remnant( ofs ) \
5844 STBIR_SIMD_NO_UNROLL(decode); \
5845 stbir__simdf_load( cs, hc + (ofs) ); \
5846 stbir__simdf_0123to0000( c, cs ); \
5847 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*7 ); \
5848 stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*7+3 ); \
5849 stbir__simdf_0123to1111( c, cs ); \
5850 stbir__simdf_madd_mem( tot2, tot2, c, decode+(ofs)*7+7 ); \
5851 stbir__simdf_madd_mem( tot3, tot3, c, decode+(ofs)*7+10 ); \
5852 stbir__simdf_0123to2222( c, cs ); \
5853 stbir__simdf_madd_mem( tot0, tot0, c, decode+(ofs)*7+14 ); \
5854 stbir__simdf_madd_mem( tot1, tot1, c, decode+(ofs)*7+17 );
5855
5856#define stbir__store_output() \
5857 stbir__simdf_add( tot0, tot0, tot2 ); \
5858 stbir__simdf_add( tot1, tot1, tot3 ); \
5859 stbir__simdf_store( output+3, tot1 ); \
5860 stbir__simdf_store( output, tot0 ); \
5861 horizontal_coefficients += coefficient_width; \
5862 ++horizontal_contributors; \
5863 output += 7;
5864
5865#endif
5866
5867#else
5868
5869#define stbir__1_coeff_only() \
5870 float tot0, tot1, tot2, tot3, tot4, tot5, tot6, c; \
5871 c = hc[0]; \
5872 tot0 = decode[0]*c; \
5873 tot1 = decode[1]*c; \
5874 tot2 = decode[2]*c; \
5875 tot3 = decode[3]*c; \
5876 tot4 = decode[4]*c; \
5877 tot5 = decode[5]*c; \
5878 tot6 = decode[6]*c;
5879
5880#define stbir__2_coeff_only() \
5881 float tot0, tot1, tot2, tot3, tot4, tot5, tot6, c; \
5882 c = hc[0]; \
5883 tot0 = decode[0]*c; \
5884 tot1 = decode[1]*c; \
5885 tot2 = decode[2]*c; \
5886 tot3 = decode[3]*c; \
5887 tot4 = decode[4]*c; \
5888 tot5 = decode[5]*c; \
5889 tot6 = decode[6]*c; \
5890 c = hc[1]; \
5891 tot0 += decode[7]*c; \
5892 tot1 += decode[8]*c; \
5893 tot2 += decode[9]*c; \
5894 tot3 += decode[10]*c; \
5895 tot4 += decode[11]*c; \
5896 tot5 += decode[12]*c; \
5897 tot6 += decode[13]*c; \
5898
5899#define stbir__3_coeff_only() \
5900 float tot0, tot1, tot2, tot3, tot4, tot5, tot6, c; \
5901 c = hc[0]; \
5902 tot0 = decode[0]*c; \
5903 tot1 = decode[1]*c; \
5904 tot2 = decode[2]*c; \
5905 tot3 = decode[3]*c; \
5906 tot4 = decode[4]*c; \
5907 tot5 = decode[5]*c; \
5908 tot6 = decode[6]*c; \
5909 c = hc[1]; \
5910 tot0 += decode[7]*c; \
5911 tot1 += decode[8]*c; \
5912 tot2 += decode[9]*c; \
5913 tot3 += decode[10]*c; \
5914 tot4 += decode[11]*c; \
5915 tot5 += decode[12]*c; \
5916 tot6 += decode[13]*c; \
5917 c = hc[2]; \
5918 tot0 += decode[14]*c; \
5919 tot1 += decode[15]*c; \
5920 tot2 += decode[16]*c; \
5921 tot3 += decode[17]*c; \
5922 tot4 += decode[18]*c; \
5923 tot5 += decode[19]*c; \
5924 tot6 += decode[20]*c; \
5925
5926#define stbir__store_output_tiny() \
5927 output[0] = tot0; \
5928 output[1] = tot1; \
5929 output[2] = tot2; \
5930 output[3] = tot3; \
5931 output[4] = tot4; \
5932 output[5] = tot5; \
5933 output[6] = tot6; \
5934 horizontal_coefficients += coefficient_width; \
5935 ++horizontal_contributors; \
5936 output += 7;
5937
5938#define stbir__4_coeff_start() \
5939 float x0,x1,x2,x3,x4,x5,x6,y0,y1,y2,y3,y4,y5,y6,c; \
5940 STBIR_SIMD_NO_UNROLL(decode); \
5941 c = hc[0]; \
5942 x0 = decode[0] * c; \
5943 x1 = decode[1] * c; \
5944 x2 = decode[2] * c; \
5945 x3 = decode[3] * c; \
5946 x4 = decode[4] * c; \
5947 x5 = decode[5] * c; \
5948 x6 = decode[6] * c; \
5949 c = hc[1]; \
5950 y0 = decode[7] * c; \
5951 y1 = decode[8] * c; \
5952 y2 = decode[9] * c; \
5953 y3 = decode[10] * c; \
5954 y4 = decode[11] * c; \
5955 y5 = decode[12] * c; \
5956 y6 = decode[13] * c; \
5957 c = hc[2]; \
5958 x0 += decode[14] * c; \
5959 x1 += decode[15] * c; \
5960 x2 += decode[16] * c; \
5961 x3 += decode[17] * c; \
5962 x4 += decode[18] * c; \
5963 x5 += decode[19] * c; \
5964 x6 += decode[20] * c; \
5965 c = hc[3]; \
5966 y0 += decode[21] * c; \
5967 y1 += decode[22] * c; \
5968 y2 += decode[23] * c; \
5969 y3 += decode[24] * c; \
5970 y4 += decode[25] * c; \
5971 y5 += decode[26] * c; \
5972 y6 += decode[27] * c;
5973
5974#define stbir__4_coeff_continue_from_4( ofs ) \
5975 STBIR_SIMD_NO_UNROLL(decode); \
5976 c = hc[0+(ofs)]; \
5977 x0 += decode[0+(ofs)*7] * c; \
5978 x1 += decode[1+(ofs)*7] * c; \
5979 x2 += decode[2+(ofs)*7] * c; \
5980 x3 += decode[3+(ofs)*7] * c; \
5981 x4 += decode[4+(ofs)*7] * c; \
5982 x5 += decode[5+(ofs)*7] * c; \
5983 x6 += decode[6+(ofs)*7] * c; \
5984 c = hc[1+(ofs)]; \
5985 y0 += decode[7+(ofs)*7] * c; \
5986 y1 += decode[8+(ofs)*7] * c; \
5987 y2 += decode[9+(ofs)*7] * c; \
5988 y3 += decode[10+(ofs)*7] * c; \
5989 y4 += decode[11+(ofs)*7] * c; \
5990 y5 += decode[12+(ofs)*7] * c; \
5991 y6 += decode[13+(ofs)*7] * c; \
5992 c = hc[2+(ofs)]; \
5993 x0 += decode[14+(ofs)*7] * c; \
5994 x1 += decode[15+(ofs)*7] * c; \
5995 x2 += decode[16+(ofs)*7] * c; \
5996 x3 += decode[17+(ofs)*7] * c; \
5997 x4 += decode[18+(ofs)*7] * c; \
5998 x5 += decode[19+(ofs)*7] * c; \
5999 x6 += decode[20+(ofs)*7] * c; \
6000 c = hc[3+(ofs)]; \
6001 y0 += decode[21+(ofs)*7] * c; \
6002 y1 += decode[22+(ofs)*7] * c; \
6003 y2 += decode[23+(ofs)*7] * c; \
6004 y3 += decode[24+(ofs)*7] * c; \
6005 y4 += decode[25+(ofs)*7] * c; \
6006 y5 += decode[26+(ofs)*7] * c; \
6007 y6 += decode[27+(ofs)*7] * c;
6008
6009#define stbir__1_coeff_remnant( ofs ) \
6010 STBIR_SIMD_NO_UNROLL(decode); \
6011 c = hc[0+(ofs)]; \
6012 x0 += decode[0+(ofs)*7] * c; \
6013 x1 += decode[1+(ofs)*7] * c; \
6014 x2 += decode[2+(ofs)*7] * c; \
6015 x3 += decode[3+(ofs)*7] * c; \
6016 x4 += decode[4+(ofs)*7] * c; \
6017 x5 += decode[5+(ofs)*7] * c; \
6018 x6 += decode[6+(ofs)*7] * c; \
6019
6020#define stbir__2_coeff_remnant( ofs ) \
6021 STBIR_SIMD_NO_UNROLL(decode); \
6022 c = hc[0+(ofs)]; \
6023 x0 += decode[0+(ofs)*7] * c; \
6024 x1 += decode[1+(ofs)*7] * c; \
6025 x2 += decode[2+(ofs)*7] * c; \
6026 x3 += decode[3+(ofs)*7] * c; \
6027 x4 += decode[4+(ofs)*7] * c; \
6028 x5 += decode[5+(ofs)*7] * c; \
6029 x6 += decode[6+(ofs)*7] * c; \
6030 c = hc[1+(ofs)]; \
6031 y0 += decode[7+(ofs)*7] * c; \
6032 y1 += decode[8+(ofs)*7] * c; \
6033 y2 += decode[9+(ofs)*7] * c; \
6034 y3 += decode[10+(ofs)*7] * c; \
6035 y4 += decode[11+(ofs)*7] * c; \
6036 y5 += decode[12+(ofs)*7] * c; \
6037 y6 += decode[13+(ofs)*7] * c; \
6038
6039#define stbir__3_coeff_remnant( ofs ) \
6040 STBIR_SIMD_NO_UNROLL(decode); \
6041 c = hc[0+(ofs)]; \
6042 x0 += decode[0+(ofs)*7] * c; \
6043 x1 += decode[1+(ofs)*7] * c; \
6044 x2 += decode[2+(ofs)*7] * c; \
6045 x3 += decode[3+(ofs)*7] * c; \
6046 x4 += decode[4+(ofs)*7] * c; \
6047 x5 += decode[5+(ofs)*7] * c; \
6048 x6 += decode[6+(ofs)*7] * c; \
6049 c = hc[1+(ofs)]; \
6050 y0 += decode[7+(ofs)*7] * c; \
6051 y1 += decode[8+(ofs)*7] * c; \
6052 y2 += decode[9+(ofs)*7] * c; \
6053 y3 += decode[10+(ofs)*7] * c; \
6054 y4 += decode[11+(ofs)*7] * c; \
6055 y5 += decode[12+(ofs)*7] * c; \
6056 y6 += decode[13+(ofs)*7] * c; \
6057 c = hc[2+(ofs)]; \
6058 x0 += decode[14+(ofs)*7] * c; \
6059 x1 += decode[15+(ofs)*7] * c; \
6060 x2 += decode[16+(ofs)*7] * c; \
6061 x3 += decode[17+(ofs)*7] * c; \
6062 x4 += decode[18+(ofs)*7] * c; \
6063 x5 += decode[19+(ofs)*7] * c; \
6064 x6 += decode[20+(ofs)*7] * c; \
6065
6066#define stbir__store_output() \
6067 output[0] = x0 + y0; \
6068 output[1] = x1 + y1; \
6069 output[2] = x2 + y2; \
6070 output[3] = x3 + y3; \
6071 output[4] = x4 + y4; \
6072 output[5] = x5 + y5; \
6073 output[6] = x6 + y6; \
6074 horizontal_coefficients += coefficient_width; \
6075 ++horizontal_contributors; \
6076 output += 7;
6077
6078#endif
6079
6080#define STBIR__horizontal_channels 7
6081#define STB_IMAGE_RESIZE_DO_HORIZONTALS
6082#include STBIR__HEADER_FILENAME
6083
6084
6085// include all of the vertical resamplers (both scatter and gather versions)
6086
6087#define STBIR__vertical_channels 1
6088#define STB_IMAGE_RESIZE_DO_VERTICALS
6089#include STBIR__HEADER_FILENAME
6090
6091#define STBIR__vertical_channels 1
6092#define STB_IMAGE_RESIZE_DO_VERTICALS
6093#define STB_IMAGE_RESIZE_VERTICAL_CONTINUE
6094#include STBIR__HEADER_FILENAME
6095
6096#define STBIR__vertical_channels 2
6097#define STB_IMAGE_RESIZE_DO_VERTICALS
6098#include STBIR__HEADER_FILENAME
6099
6100#define STBIR__vertical_channels 2
6101#define STB_IMAGE_RESIZE_DO_VERTICALS
6102#define STB_IMAGE_RESIZE_VERTICAL_CONTINUE
6103#include STBIR__HEADER_FILENAME
6104
6105#define STBIR__vertical_channels 3
6106#define STB_IMAGE_RESIZE_DO_VERTICALS
6107#include STBIR__HEADER_FILENAME
6108
6109#define STBIR__vertical_channels 3
6110#define STB_IMAGE_RESIZE_DO_VERTICALS
6111#define STB_IMAGE_RESIZE_VERTICAL_CONTINUE
6112#include STBIR__HEADER_FILENAME
6113
6114#define STBIR__vertical_channels 4
6115#define STB_IMAGE_RESIZE_DO_VERTICALS
6116#include STBIR__HEADER_FILENAME
6117
6118#define STBIR__vertical_channels 4
6119#define STB_IMAGE_RESIZE_DO_VERTICALS
6120#define STB_IMAGE_RESIZE_VERTICAL_CONTINUE
6121#include STBIR__HEADER_FILENAME
6122
6123#define STBIR__vertical_channels 5
6124#define STB_IMAGE_RESIZE_DO_VERTICALS
6125#include STBIR__HEADER_FILENAME
6126
6127#define STBIR__vertical_channels 5
6128#define STB_IMAGE_RESIZE_DO_VERTICALS
6129#define STB_IMAGE_RESIZE_VERTICAL_CONTINUE
6130#include STBIR__HEADER_FILENAME
6131
6132#define STBIR__vertical_channels 6
6133#define STB_IMAGE_RESIZE_DO_VERTICALS
6134#include STBIR__HEADER_FILENAME
6135
6136#define STBIR__vertical_channels 6
6137#define STB_IMAGE_RESIZE_DO_VERTICALS
6138#define STB_IMAGE_RESIZE_VERTICAL_CONTINUE
6139#include STBIR__HEADER_FILENAME
6140
6141#define STBIR__vertical_channels 7
6142#define STB_IMAGE_RESIZE_DO_VERTICALS
6143#include STBIR__HEADER_FILENAME
6144
6145#define STBIR__vertical_channels 7
6146#define STB_IMAGE_RESIZE_DO_VERTICALS
6147#define STB_IMAGE_RESIZE_VERTICAL_CONTINUE
6148#include STBIR__HEADER_FILENAME
6149
6150#define STBIR__vertical_channels 8
6151#define STB_IMAGE_RESIZE_DO_VERTICALS
6152#include STBIR__HEADER_FILENAME
6153
6154#define STBIR__vertical_channels 8
6155#define STB_IMAGE_RESIZE_DO_VERTICALS
6156#define STB_IMAGE_RESIZE_VERTICAL_CONTINUE
6157#include STBIR__HEADER_FILENAME
6158
6159typedef void STBIR_VERTICAL_GATHERFUNC( float * output, float const * coeffs, float const ** inputs, float const * input0_end );
6160
6161static STBIR_VERTICAL_GATHERFUNC * stbir__vertical_gathers[ 8 ] =
6162{
6163 stbir__vertical_gather_with_1_coeffs,stbir__vertical_gather_with_2_coeffs,stbir__vertical_gather_with_3_coeffs,stbir__vertical_gather_with_4_coeffs,stbir__vertical_gather_with_5_coeffs,stbir__vertical_gather_with_6_coeffs,stbir__vertical_gather_with_7_coeffs,stbir__vertical_gather_with_8_coeffs
6164};
6165
6166static STBIR_VERTICAL_GATHERFUNC * stbir__vertical_gathers_continues[ 8 ] =
6167{
6168 stbir__vertical_gather_with_1_coeffs_cont,stbir__vertical_gather_with_2_coeffs_cont,stbir__vertical_gather_with_3_coeffs_cont,stbir__vertical_gather_with_4_coeffs_cont,stbir__vertical_gather_with_5_coeffs_cont,stbir__vertical_gather_with_6_coeffs_cont,stbir__vertical_gather_with_7_coeffs_cont,stbir__vertical_gather_with_8_coeffs_cont
6169};
6170
6171typedef void STBIR_VERTICAL_SCATTERFUNC( float ** outputs, float const * coeffs, float const * input, float const * input_end );
6172
6173static STBIR_VERTICAL_SCATTERFUNC * stbir__vertical_scatter_sets[ 8 ] =
6174{
6175 stbir__vertical_scatter_with_1_coeffs,stbir__vertical_scatter_with_2_coeffs,stbir__vertical_scatter_with_3_coeffs,stbir__vertical_scatter_with_4_coeffs,stbir__vertical_scatter_with_5_coeffs,stbir__vertical_scatter_with_6_coeffs,stbir__vertical_scatter_with_7_coeffs,stbir__vertical_scatter_with_8_coeffs
6176};
6177
6178static STBIR_VERTICAL_SCATTERFUNC * stbir__vertical_scatter_blends[ 8 ] =
6179{
6180 stbir__vertical_scatter_with_1_coeffs_cont,stbir__vertical_scatter_with_2_coeffs_cont,stbir__vertical_scatter_with_3_coeffs_cont,stbir__vertical_scatter_with_4_coeffs_cont,stbir__vertical_scatter_with_5_coeffs_cont,stbir__vertical_scatter_with_6_coeffs_cont,stbir__vertical_scatter_with_7_coeffs_cont,stbir__vertical_scatter_with_8_coeffs_cont
6181};
6182
6183
6184static void stbir__encode_scanline( stbir__info const * stbir_info, void *output_buffer_data, float * encode_buffer, int row STBIR_ONLY_PROFILE_GET_SPLIT_INFO )
6185{
6186 int num_pixels = stbir_info->horizontal.scale_info.output_sub_size;
6187 int channels = stbir_info->channels;
6188 int width_times_channels = num_pixels * channels;
6189 void * output_buffer;
6190
6191 // un-alpha weight if we need to
6192 if ( stbir_info->alpha_unweight )
6193 {
6194 STBIR_PROFILE_START( unalpha );
6195 stbir_info->alpha_unweight( encode_buffer, width_times_channels );
6196 STBIR_PROFILE_END( unalpha );
6197 }
6198
6199 // write directly into output by default
6200 output_buffer = output_buffer_data;
6201
6202 // if we have an output callback, we first convert the decode buffer in place (and then hand that to the callback)
6203 if ( stbir_info->out_pixels_cb )
6204 output_buffer = encode_buffer;
6205
6206 STBIR_PROFILE_START( encode );
6207 // convert into the output buffer
6208 stbir_info->encode_pixels( output_buffer, width_times_channels, encode_buffer );
6209 STBIR_PROFILE_END( encode );
6210
6211 // if we have an output callback, call it to send the data
6212 if ( stbir_info->out_pixels_cb )
6213 stbir_info->out_pixels_cb( output_buffer, num_pixels, row, stbir_info->user_data );
6214}
6215
6216
6217// Get the ring buffer pointer for an index
6218static float* stbir__get_ring_buffer_entry(stbir__info const * stbir_info, stbir__per_split_info const * split_info, int index )
6219{
6220 STBIR_ASSERT( index < stbir_info->ring_buffer_num_entries );
6221
6222 #ifdef STBIR__SEPARATE_ALLOCATIONS
6223 return split_info->ring_buffers[ index ];
6224 #else
6225 return (float*) ( ( (char*) split_info->ring_buffer ) + ( index * stbir_info->ring_buffer_length_bytes ) );
6226 #endif
6227}
6228
6229// Get the specified scan line from the ring buffer
6230static float* stbir__get_ring_buffer_scanline(stbir__info const * stbir_info, stbir__per_split_info const * split_info, int get_scanline)
6231{
6232 int ring_buffer_index = (split_info->ring_buffer_begin_index + (get_scanline - split_info->ring_buffer_first_scanline)) % stbir_info->ring_buffer_num_entries;
6233 return stbir__get_ring_buffer_entry( stbir_info, split_info, ring_buffer_index );
6234}
6235
6236static void stbir__resample_horizontal_gather(stbir__info const * stbir_info, float* output_buffer, float const * input_buffer STBIR_ONLY_PROFILE_GET_SPLIT_INFO )
6237{
6238 float const * decode_buffer = input_buffer - ( stbir_info->scanline_extents.conservative.n0 * stbir_info->effective_channels );
6239
6240 STBIR_PROFILE_START( horizontal );
6241 if ( ( stbir_info->horizontal.filter_enum == STBIR_FILTER_POINT_SAMPLE ) && ( stbir_info->horizontal.scale_info.scale == 1.0f ) )
6242 STBIR_MEMCPY( output_buffer, input_buffer, stbir_info->horizontal.scale_info.output_sub_size * sizeof( float ) * stbir_info->effective_channels );
6243 else
6244 stbir_info->horizontal_gather_channels( output_buffer, stbir_info->horizontal.scale_info.output_sub_size, decode_buffer, stbir_info->horizontal.contributors, stbir_info->horizontal.coefficients, stbir_info->horizontal.coefficient_width );
6245 STBIR_PROFILE_END( horizontal );
6246}
6247
6248static void stbir__resample_vertical_gather(stbir__info const * stbir_info, stbir__per_split_info* split_info, int n, int contrib_n0, int contrib_n1, float const * vertical_coefficients )
6249{
6250 float* encode_buffer = split_info->vertical_buffer;
6251 float* decode_buffer = split_info->decode_buffer;
6252 int vertical_first = stbir_info->vertical_first;
6253 int width = (vertical_first) ? ( stbir_info->scanline_extents.conservative.n1-stbir_info->scanline_extents.conservative.n0+1 ) : stbir_info->horizontal.scale_info.output_sub_size;
6254 int width_times_channels = stbir_info->effective_channels * width;
6255
6256 STBIR_ASSERT( stbir_info->vertical.is_gather );
6257
6258 // loop over the contributing scanlines and scale into the buffer
6259 STBIR_PROFILE_START( vertical );
6260 {
6261 int k = 0, total = contrib_n1 - contrib_n0 + 1;
6262 STBIR_ASSERT( total > 0 );
6263 do {
6264 float const * inputs[8];
6265 int i, cnt = total; if ( cnt > 8 ) cnt = 8;
6266 for( i = 0 ; i < cnt ; i++ )
6267 inputs[ i ] = stbir__get_ring_buffer_scanline(stbir_info, split_info, k+i+contrib_n0 );
6268
6269 // call the N scanlines at a time function (up to 8 scanlines of blending at once)
6270 ((k==0)?stbir__vertical_gathers:stbir__vertical_gathers_continues)[cnt-1]( (vertical_first) ? decode_buffer : encode_buffer, vertical_coefficients + k, inputs, inputs[0] + width_times_channels );
6271 k += cnt;
6272 total -= cnt;
6273 } while ( total );
6274 }
6275 STBIR_PROFILE_END( vertical );
6276
6277 if ( vertical_first )
6278 {
6279 // Now resample the gathered vertical data in the horizontal axis into the encode buffer
6280 decode_buffer[ width_times_channels ] = 0.0f; // clear two over for horizontals with a remnant of 3
6281 decode_buffer[ width_times_channels+1 ] = 0.0f;
6282 stbir__resample_horizontal_gather(stbir_info, encode_buffer, decode_buffer STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
6283 }
6284
6285 stbir__encode_scanline( stbir_info, ( (char *) stbir_info->output_data ) + ((size_t)n * (size_t)stbir_info->output_stride_bytes),
6286 encode_buffer, n STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
6287}
6288
6289static void stbir__decode_and_resample_for_vertical_gather_loop(stbir__info const * stbir_info, stbir__per_split_info* split_info, int n)
6290{
6291 int ring_buffer_index;
6292 float* ring_buffer;
6293
6294 // Decode the nth scanline from the source image into the decode buffer.
6295 stbir__decode_scanline( stbir_info, n, split_info->decode_buffer STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
6296
6297 // update new end scanline
6298 split_info->ring_buffer_last_scanline = n;
6299
6300 // get ring buffer
6301 ring_buffer_index = (split_info->ring_buffer_begin_index + (split_info->ring_buffer_last_scanline - split_info->ring_buffer_first_scanline)) % stbir_info->ring_buffer_num_entries;
6302 ring_buffer = stbir__get_ring_buffer_entry(stbir_info, split_info, ring_buffer_index);
6303
6304 // Now resample it into the ring buffer.
6305 stbir__resample_horizontal_gather( stbir_info, ring_buffer, split_info->decode_buffer STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
6306
6307 // Now it's sitting in the ring buffer ready to be used as source for the vertical sampling.
6308}
6309
6310static void stbir__vertical_gather_loop( stbir__info const * stbir_info, stbir__per_split_info* split_info, int split_count )
6311{
6312 int y, start_output_y, end_output_y;
6313 stbir__contributors* vertical_contributors = stbir_info->vertical.contributors;
6314 float const * vertical_coefficients = stbir_info->vertical.coefficients;
6315
6316 STBIR_ASSERT( stbir_info->vertical.is_gather );
6317
6318 start_output_y = split_info->start_output_y;
6319 end_output_y = split_info[split_count-1].end_output_y;
6320
6321 vertical_contributors += start_output_y;
6322 vertical_coefficients += start_output_y * stbir_info->vertical.coefficient_width;
6323
6324 // initialize the ring buffer for gathering
6325 split_info->ring_buffer_begin_index = 0;
6326 split_info->ring_buffer_first_scanline = vertical_contributors->n0;
6327 split_info->ring_buffer_last_scanline = split_info->ring_buffer_first_scanline - 1; // means "empty"
6328
6329 for (y = start_output_y; y < end_output_y; y++)
6330 {
6331 int in_first_scanline, in_last_scanline;
6332
6333 in_first_scanline = vertical_contributors->n0;
6334 in_last_scanline = vertical_contributors->n1;
6335
6336 // make sure the indexing hasn't broken
6337 STBIR_ASSERT( in_first_scanline >= split_info->ring_buffer_first_scanline );
6338
6339 // Load in new scanlines
6340 while (in_last_scanline > split_info->ring_buffer_last_scanline)
6341 {
6342 STBIR_ASSERT( ( split_info->ring_buffer_last_scanline - split_info->ring_buffer_first_scanline + 1 ) <= stbir_info->ring_buffer_num_entries );
6343
6344 // make sure there was room in the ring buffer when we add new scanlines
6345 if ( ( split_info->ring_buffer_last_scanline - split_info->ring_buffer_first_scanline + 1 ) == stbir_info->ring_buffer_num_entries )
6346 {
6347 split_info->ring_buffer_first_scanline++;
6348 split_info->ring_buffer_begin_index++;
6349 }
6350
6351 if ( stbir_info->vertical_first )
6352 {
6353 float * ring_buffer = stbir__get_ring_buffer_scanline( stbir_info, split_info, ++split_info->ring_buffer_last_scanline );
6354 // Decode the nth scanline from the source image into the decode buffer.
6355 stbir__decode_scanline( stbir_info, split_info->ring_buffer_last_scanline, ring_buffer STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
6356 }
6357 else
6358 {
6359 stbir__decode_and_resample_for_vertical_gather_loop(stbir_info, split_info, split_info->ring_buffer_last_scanline + 1);
6360 }
6361 }
6362
6363 // Now all buffers should be ready to write a row of vertical sampling, so do it.
6364 stbir__resample_vertical_gather(stbir_info, split_info, y, in_first_scanline, in_last_scanline, vertical_coefficients );
6365
6366 ++vertical_contributors;
6367 vertical_coefficients += stbir_info->vertical.coefficient_width;
6368 }
6369}
6370
6371#define STBIR__FLOAT_EMPTY_MARKER 3.0e+38F
6372#define STBIR__FLOAT_BUFFER_IS_EMPTY(ptr) ((ptr)[0]==STBIR__FLOAT_EMPTY_MARKER)
6373
6374static void stbir__encode_first_scanline_from_scatter(stbir__info const * stbir_info, stbir__per_split_info* split_info)
6375{
6376 // evict a scanline out into the output buffer
6377 float* ring_buffer_entry = stbir__get_ring_buffer_entry(stbir_info, split_info, split_info->ring_buffer_begin_index );
6378
6379 // dump the scanline out
6380 stbir__encode_scanline( stbir_info, ( (char *)stbir_info->output_data ) + ( (size_t)split_info->ring_buffer_first_scanline * (size_t)stbir_info->output_stride_bytes ), ring_buffer_entry, split_info->ring_buffer_first_scanline STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
6381
6382 // mark it as empty
6383 ring_buffer_entry[ 0 ] = STBIR__FLOAT_EMPTY_MARKER;
6384
6385 // advance the first scanline
6386 split_info->ring_buffer_first_scanline++;
6387 if ( ++split_info->ring_buffer_begin_index == stbir_info->ring_buffer_num_entries )
6388 split_info->ring_buffer_begin_index = 0;
6389}
6390
6391static void stbir__horizontal_resample_and_encode_first_scanline_from_scatter(stbir__info const * stbir_info, stbir__per_split_info* split_info)
6392{
6393 // evict a scanline out into the output buffer
6394
6395 float* ring_buffer_entry = stbir__get_ring_buffer_entry(stbir_info, split_info, split_info->ring_buffer_begin_index );
6396
6397 // Now resample it into the buffer.
6398 stbir__resample_horizontal_gather( stbir_info, split_info->vertical_buffer, ring_buffer_entry STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
6399
6400 // dump the scanline out
6401 stbir__encode_scanline( stbir_info, ( (char *)stbir_info->output_data ) + ( (size_t)split_info->ring_buffer_first_scanline * (size_t)stbir_info->output_stride_bytes ), split_info->vertical_buffer, split_info->ring_buffer_first_scanline STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
6402
6403 // mark it as empty
6404 ring_buffer_entry[ 0 ] = STBIR__FLOAT_EMPTY_MARKER;
6405
6406 // advance the first scanline
6407 split_info->ring_buffer_first_scanline++;
6408 if ( ++split_info->ring_buffer_begin_index == stbir_info->ring_buffer_num_entries )
6409 split_info->ring_buffer_begin_index = 0;
6410}
6411
6412static void stbir__resample_vertical_scatter(stbir__info const * stbir_info, stbir__per_split_info* split_info, int n0, int n1, float const * vertical_coefficients, float const * vertical_buffer, float const * vertical_buffer_end )
6413{
6414 STBIR_ASSERT( !stbir_info->vertical.is_gather );
6415
6416 STBIR_PROFILE_START( vertical );
6417 {
6418 int k = 0, total = n1 - n0 + 1;
6419 STBIR_ASSERT( total > 0 );
6420 do {
6421 float * outputs[8];
6422 int i, n = total; if ( n > 8 ) n = 8;
6423 for( i = 0 ; i < n ; i++ )
6424 {
6425 outputs[ i ] = stbir__get_ring_buffer_scanline(stbir_info, split_info, k+i+n0 );
6426 if ( ( i ) && ( STBIR__FLOAT_BUFFER_IS_EMPTY( outputs[i] ) != STBIR__FLOAT_BUFFER_IS_EMPTY( outputs[0] ) ) ) // make sure runs are of the same type
6427 {
6428 n = i;
6429 break;
6430 }
6431 }
6432 // call the scatter to N scanlines at a time function (up to 8 scanlines of scattering at once)
6433 ((STBIR__FLOAT_BUFFER_IS_EMPTY( outputs[0] ))?stbir__vertical_scatter_sets:stbir__vertical_scatter_blends)[n-1]( outputs, vertical_coefficients + k, vertical_buffer, vertical_buffer_end );
6434 k += n;
6435 total -= n;
6436 } while ( total );
6437 }
6438
6439 STBIR_PROFILE_END( vertical );
6440}
6441
6442typedef void stbir__handle_scanline_for_scatter_func(stbir__info const * stbir_info, stbir__per_split_info* split_info);
6443
6444static void stbir__vertical_scatter_loop( stbir__info const * stbir_info, stbir__per_split_info* split_info, int split_count )
6445{
6446 int y, start_output_y, end_output_y, start_input_y, end_input_y;
6447 stbir__contributors* vertical_contributors = stbir_info->vertical.contributors;
6448 float const * vertical_coefficients = stbir_info->vertical.coefficients;
6449 stbir__handle_scanline_for_scatter_func * handle_scanline_for_scatter;
6450 void * scanline_scatter_buffer;
6451 void * scanline_scatter_buffer_end;
6452 int on_first_input_y, last_input_y;
6453 int width = (stbir_info->vertical_first) ? ( stbir_info->scanline_extents.conservative.n1-stbir_info->scanline_extents.conservative.n0+1 ) : stbir_info->horizontal.scale_info.output_sub_size;
6454 int width_times_channels = stbir_info->effective_channels * width;
6455
6456 STBIR_ASSERT( !stbir_info->vertical.is_gather );
6457
6458 start_output_y = split_info->start_output_y;
6459 end_output_y = split_info[split_count-1].end_output_y; // may do multiple split counts
6460
6461 start_input_y = split_info->start_input_y;
6462 end_input_y = split_info[split_count-1].end_input_y;
6463
6464 // adjust for starting offset start_input_y
6465 y = start_input_y + stbir_info->vertical.filter_pixel_margin;
6466 vertical_contributors += y ;
6467 vertical_coefficients += stbir_info->vertical.coefficient_width * y;
6468
6469 if ( stbir_info->vertical_first )
6470 {
6471 handle_scanline_for_scatter = stbir__horizontal_resample_and_encode_first_scanline_from_scatter;
6472 scanline_scatter_buffer = split_info->decode_buffer;
6473 scanline_scatter_buffer_end = ( (char*) scanline_scatter_buffer ) + sizeof( float ) * stbir_info->effective_channels * (stbir_info->scanline_extents.conservative.n1-stbir_info->scanline_extents.conservative.n0+1);
6474 }
6475 else
6476 {
6477 handle_scanline_for_scatter = stbir__encode_first_scanline_from_scatter;
6478 scanline_scatter_buffer = split_info->vertical_buffer;
6479 scanline_scatter_buffer_end = ( (char*) scanline_scatter_buffer ) + sizeof( float ) * stbir_info->effective_channels * stbir_info->horizontal.scale_info.output_sub_size;
6480 }
6481
6482 // initialize the ring buffer for scattering
6483 split_info->ring_buffer_first_scanline = start_output_y;
6484 split_info->ring_buffer_last_scanline = -1;
6485 split_info->ring_buffer_begin_index = -1;
6486
6487 // mark all the buffers as empty to start
6488 for( y = 0 ; y < stbir_info->ring_buffer_num_entries ; y++ )
6489 {
6490 float * decode_buffer = stbir__get_ring_buffer_entry( stbir_info, split_info, y );
6491 decode_buffer[ width_times_channels ] = 0.0f; // clear two over for horizontals with a remnant of 3
6492 decode_buffer[ width_times_channels+1 ] = 0.0f;
6493 decode_buffer[0] = STBIR__FLOAT_EMPTY_MARKER; // only used on scatter
6494 }
6495
6496 // do the loop in input space
6497 on_first_input_y = 1; last_input_y = start_input_y;
6498 for (y = start_input_y ; y < end_input_y; y++)
6499 {
6500 int out_first_scanline, out_last_scanline;
6501
6502 out_first_scanline = vertical_contributors->n0;
6503 out_last_scanline = vertical_contributors->n1;
6504
6505 STBIR_ASSERT(out_last_scanline - out_first_scanline + 1 <= stbir_info->ring_buffer_num_entries);
6506
6507 if ( ( out_last_scanline >= out_first_scanline ) && ( ( ( out_first_scanline >= start_output_y ) && ( out_first_scanline < end_output_y ) ) || ( ( out_last_scanline >= start_output_y ) && ( out_last_scanline < end_output_y ) ) ) )
6508 {
6509 float const * vc = vertical_coefficients;
6510
6511 // keep track of the range actually seen for the next resize
6512 last_input_y = y;
6513 if ( ( on_first_input_y ) && ( y > start_input_y ) )
6514 split_info->start_input_y = y;
6515 on_first_input_y = 0;
6516
6517 // clip the region
6518 if ( out_first_scanline < start_output_y )
6519 {
6520 vc += start_output_y - out_first_scanline;
6521 out_first_scanline = start_output_y;
6522 }
6523
6524 if ( out_last_scanline >= end_output_y )
6525 out_last_scanline = end_output_y - 1;
6526
6527 // if very first scanline, init the index
6528 if (split_info->ring_buffer_begin_index < 0)
6529 split_info->ring_buffer_begin_index = out_first_scanline - start_output_y;
6530
6531 STBIR_ASSERT( split_info->ring_buffer_begin_index <= out_first_scanline );
6532
6533 // Decode the nth scanline from the source image into the decode buffer.
6534 stbir__decode_scanline( stbir_info, y, split_info->decode_buffer STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
6535
6536 // When horizontal first, we resample horizontally into the vertical buffer before we scatter it out
6537 if ( !stbir_info->vertical_first )
6538 stbir__resample_horizontal_gather( stbir_info, split_info->vertical_buffer, split_info->decode_buffer STBIR_ONLY_PROFILE_SET_SPLIT_INFO );
6539
6540 // Now it's sitting in the buffer ready to be distributed into the ring buffers.
6541
6542 // evict from the ringbuffer, if we need are full
6543 if ( ( ( split_info->ring_buffer_last_scanline - split_info->ring_buffer_first_scanline + 1 ) == stbir_info->ring_buffer_num_entries ) &&
6544 ( out_last_scanline > split_info->ring_buffer_last_scanline ) )
6545 handle_scanline_for_scatter( stbir_info, split_info );
6546
6547 // Now the horizontal buffer is ready to write to all ring buffer rows, so do it.
6548 stbir__resample_vertical_scatter(stbir_info, split_info, out_first_scanline, out_last_scanline, vc, (float*)scanline_scatter_buffer, (float*)scanline_scatter_buffer_end );
6549
6550 // update the end of the buffer
6551 if ( out_last_scanline > split_info->ring_buffer_last_scanline )
6552 split_info->ring_buffer_last_scanline = out_last_scanline;
6553 }
6554 ++vertical_contributors;
6555 vertical_coefficients += stbir_info->vertical.coefficient_width;
6556 }
6557
6558 // now evict the scanlines that are left over in the ring buffer
6559 while ( split_info->ring_buffer_first_scanline < end_output_y )
6560 handle_scanline_for_scatter(stbir_info, split_info);
6561
6562 // update the end_input_y if we do multiple resizes with the same data
6563 ++last_input_y;
6564 for( y = 0 ; y < split_count; y++ )
6565 if ( split_info[y].end_input_y > last_input_y )
6566 split_info[y].end_input_y = last_input_y;
6567}
6568
6569
6570static stbir__kernel_callback * stbir__builtin_kernels[] = { 0, stbir__filter_trapezoid, stbir__filter_triangle, stbir__filter_cubic, stbir__filter_catmullrom, stbir__filter_mitchell, stbir__filter_point };
6571static stbir__support_callback * stbir__builtin_supports[] = { 0, stbir__support_trapezoid, stbir__support_one, stbir__support_two, stbir__support_two, stbir__support_two, stbir__support_zeropoint5 };
6572
6573static void stbir__set_sampler(stbir__sampler * samp, stbir_filter filter, stbir__kernel_callback * kernel, stbir__support_callback * support, stbir_edge edge, stbir__scale_info * scale_info, int always_gather, void * user_data )
6574{
6575 // set filter
6576 if (filter == 0)
6577 {
6578 filter = STBIR_DEFAULT_FILTER_DOWNSAMPLE; // default to downsample
6579 if (scale_info->scale >= ( 1.0f - stbir__small_float ) )
6580 {
6581 if ( (scale_info->scale <= ( 1.0f + stbir__small_float ) ) && ( STBIR_CEILF(scale_info->pixel_shift) == scale_info->pixel_shift ) )
6582 filter = STBIR_FILTER_POINT_SAMPLE;
6583 else
6584 filter = STBIR_DEFAULT_FILTER_UPSAMPLE;
6585 }
6586 }
6587 samp->filter_enum = filter;
6588
6589 STBIR_ASSERT(samp->filter_enum != 0);
6590 STBIR_ASSERT((unsigned)samp->filter_enum < STBIR_FILTER_OTHER);
6591 samp->filter_kernel = stbir__builtin_kernels[ filter ];
6592 samp->filter_support = stbir__builtin_supports[ filter ];
6593
6594 if ( kernel && support )
6595 {
6596 samp->filter_kernel = kernel;
6597 samp->filter_support = support;
6598 samp->filter_enum = STBIR_FILTER_OTHER;
6599 }
6600
6601 samp->edge = edge;
6602 samp->filter_pixel_width = stbir__get_filter_pixel_width (samp->filter_support, scale_info->scale, user_data );
6603 // Gather is always better, but in extreme downsamples, you have to most or all of the data in memory
6604 // For horizontal, we always have all the pixels, so we always use gather here (always_gather==1).
6605 // For vertical, we use gather if scaling up (which means we will have samp->filter_pixel_width
6606 // scanlines in memory at once).
6607 samp->is_gather = 0;
6608 if ( scale_info->scale >= ( 1.0f - stbir__small_float ) )
6609 samp->is_gather = 1;
6610 else if ( ( always_gather ) || ( samp->filter_pixel_width <= STBIR_FORCE_GATHER_FILTER_SCANLINES_AMOUNT ) )
6611 samp->is_gather = 2;
6612
6613 // pre calculate stuff based on the above
6614 samp->coefficient_width = stbir__get_coefficient_width(samp, samp->is_gather, user_data);
6615
6616 // filter_pixel_width is the conservative size in pixels of input that affect an output pixel.
6617 // In rare cases (only with 2 pix to 1 pix with the default filters), it's possible that the
6618 // filter will extend before or after the scanline beyond just one extra entire copy of the
6619 // scanline (we would hit the edge twice). We don't let you do that, so we clamp the total
6620 // width to 3x the total of input pixel (once for the scanline, once for the left side
6621 // overhang, and once for the right side). We only do this for edge mode, since the other
6622 // modes can just re-edge clamp back in again.
6623 if ( edge == STBIR_EDGE_WRAP )
6624 if ( samp->filter_pixel_width > ( scale_info->input_full_size * 3 ) )
6625 samp->filter_pixel_width = scale_info->input_full_size * 3;
6626
6627 // This is how much to expand buffers to account for filters seeking outside
6628 // the image boundaries.
6629 samp->filter_pixel_margin = samp->filter_pixel_width / 2;
6630
6631 // filter_pixel_margin is the amount that this filter can overhang on just one side of either
6632 // end of the scanline (left or the right). Since we only allow you to overhang 1 scanline's
6633 // worth of pixels, we clamp this one side of overhang to the input scanline size. Again,
6634 // this clamping only happens in rare cases with the default filters (2 pix to 1 pix).
6635 if ( edge == STBIR_EDGE_WRAP )
6636 if ( samp->filter_pixel_margin > scale_info->input_full_size )
6637 samp->filter_pixel_margin = scale_info->input_full_size;
6638
6639 samp->num_contributors = stbir__get_contributors(samp, samp->is_gather);
6640
6641 samp->contributors_size = samp->num_contributors * sizeof(stbir__contributors);
6642 samp->coefficients_size = samp->num_contributors * samp->coefficient_width * sizeof(float) + sizeof(float)*STBIR_INPUT_CALLBACK_PADDING; // extra sizeof(float) is padding
6643
6644 samp->gather_prescatter_contributors = 0;
6645 samp->gather_prescatter_coefficients = 0;
6646 if ( samp->is_gather == 0 )
6647 {
6648 samp->gather_prescatter_coefficient_width = samp->filter_pixel_width;
6649 samp->gather_prescatter_num_contributors = stbir__get_contributors(samp, 2);
6650 samp->gather_prescatter_contributors_size = samp->gather_prescatter_num_contributors * sizeof(stbir__contributors);
6651 samp->gather_prescatter_coefficients_size = samp->gather_prescatter_num_contributors * samp->gather_prescatter_coefficient_width * sizeof(float);
6652 }
6653}
6654
6655static void stbir__get_conservative_extents( stbir__sampler * samp, stbir__contributors * range, void * user_data )
6656{
6657 float scale = samp->scale_info.scale;
6658 float out_shift = samp->scale_info.pixel_shift;
6659 stbir__support_callback * support = samp->filter_support;
6660 int input_full_size = samp->scale_info.input_full_size;
6661 stbir_edge edge = samp->edge;
6662 float inv_scale = samp->scale_info.inv_scale;
6663
6664 STBIR_ASSERT( samp->is_gather != 0 );
6665
6666 if ( samp->is_gather == 1 )
6667 {
6668 int in_first_pixel, in_last_pixel;
6669 float out_filter_radius = support(inv_scale, user_data) * scale;
6670
6671 stbir__calculate_in_pixel_range( &in_first_pixel, &in_last_pixel, 0.5, out_filter_radius, inv_scale, out_shift, input_full_size, edge );
6672 range->n0 = in_first_pixel;
6673 stbir__calculate_in_pixel_range( &in_first_pixel, &in_last_pixel, ( (float)(samp->scale_info.output_sub_size-1) ) + 0.5f, out_filter_radius, inv_scale, out_shift, input_full_size, edge );
6674 range->n1 = in_last_pixel;
6675 }
6676 else if ( samp->is_gather == 2 ) // downsample gather, refine
6677 {
6678 float in_pixels_radius = support(scale, user_data) * inv_scale;
6679 int filter_pixel_margin = samp->filter_pixel_margin;
6680 int output_sub_size = samp->scale_info.output_sub_size;
6681 int input_end;
6682 int n;
6683 int in_first_pixel, in_last_pixel;
6684
6685 // get a conservative area of the input range
6686 stbir__calculate_in_pixel_range( &in_first_pixel, &in_last_pixel, 0, 0, inv_scale, out_shift, input_full_size, edge );
6687 range->n0 = in_first_pixel;
6688 stbir__calculate_in_pixel_range( &in_first_pixel, &in_last_pixel, (float)output_sub_size, 0, inv_scale, out_shift, input_full_size, edge );
6689 range->n1 = in_last_pixel;
6690
6691 // now go through the margin to the start of area to find bottom
6692 n = range->n0 + 1;
6693 input_end = -filter_pixel_margin;
6694 while( n >= input_end )
6695 {
6696 int out_first_pixel, out_last_pixel;
6697 stbir__calculate_out_pixel_range( &out_first_pixel, &out_last_pixel, ((float)n)+0.5f, in_pixels_radius, scale, out_shift, output_sub_size );
6698 if ( out_first_pixel > out_last_pixel )
6699 break;
6700
6701 if ( ( out_first_pixel < output_sub_size ) || ( out_last_pixel >= 0 ) )
6702 range->n0 = n;
6703 --n;
6704 }
6705
6706 // now go through the end of the area through the margin to find top
6707 n = range->n1 - 1;
6708 input_end = n + 1 + filter_pixel_margin;
6709 while( n <= input_end )
6710 {
6711 int out_first_pixel, out_last_pixel;
6712 stbir__calculate_out_pixel_range( &out_first_pixel, &out_last_pixel, ((float)n)+0.5f, in_pixels_radius, scale, out_shift, output_sub_size );
6713 if ( out_first_pixel > out_last_pixel )
6714 break;
6715 if ( ( out_first_pixel < output_sub_size ) || ( out_last_pixel >= 0 ) )
6716 range->n1 = n;
6717 ++n;
6718 }
6719 }
6720
6721 if ( samp->edge == STBIR_EDGE_WRAP )
6722 {
6723 // if we are wrapping, and we are very close to the image size (so the edges might merge), just use the scanline up to the edge
6724 if ( ( range->n0 > 0 ) && ( range->n1 >= input_full_size ) )
6725 {
6726 int marg = range->n1 - input_full_size + 1;
6727 if ( ( marg + STBIR__MERGE_RUNS_PIXEL_THRESHOLD ) >= range->n0 )
6728 range->n0 = 0;
6729 }
6730 if ( ( range->n0 < 0 ) && ( range->n1 < (input_full_size-1) ) )
6731 {
6732 int marg = -range->n0;
6733 if ( ( input_full_size - marg - STBIR__MERGE_RUNS_PIXEL_THRESHOLD - 1 ) <= range->n1 )
6734 range->n1 = input_full_size - 1;
6735 }
6736 }
6737 else
6738 {
6739 // for non-edge-wrap modes, we never read over the edge, so clamp
6740 if ( range->n0 < 0 )
6741 range->n0 = 0;
6742 if ( range->n1 >= input_full_size )
6743 range->n1 = input_full_size - 1;
6744 }
6745}
6746
6747static void stbir__get_split_info( stbir__per_split_info* split_info, int splits, int output_height, int vertical_pixel_margin, int input_full_height, int is_gather, stbir__contributors * contribs )
6748{
6749 int i, cur;
6750 int left = output_height;
6751
6752 cur = 0;
6753 for( i = 0 ; i < splits ; i++ )
6754 {
6755 int each;
6756
6757 split_info[i].start_output_y = cur;
6758 each = left / ( splits - i );
6759 split_info[i].end_output_y = cur + each;
6760
6761 // ok, when we are gathering, we need to make sure we are starting on a y offset that doesn't have
6762 // a "special" set of coefficients. Basically, with exactly the right filter at exactly the right
6763 // resize at exactly the right phase, some of the coefficents can be zero. When they are zero, we
6764 // don't process them at all. But this leads to a tricky thing with the thread splits, where we
6765 // might have a set of two coeffs like this for example: (4,4) and (3,6). The 4,4 means there was
6766 // just one single coeff because things worked out perfectly (normally, they all have 4 coeffs
6767 // like the range 3,6. The problem is that if we start right on the (4,4) on a brand new thread,
6768 // then when we get to (3,6), we don't have the "3" sample in memory (because we didn't load
6769 // it on the initial (4,4) range because it didn't have a 3 (we only add new samples that are
6770 // larger than our existing samples - it's just how the eviction works). So, our solution here
6771 // is pretty simple, if we start right on a range that has samples that start earlier, then we
6772 // simply bump up our previous thread split range to include it, and then start this threads
6773 // range with the smaller sample. It just moves one scanline from one thread split to another,
6774 // so that we end with the unusual one, instead of start with it. To do this, we check 2-4
6775 // sample at each thread split start and then occassionally move them.
6776
6777 if ( ( is_gather ) && ( i ) )
6778 {
6779 stbir__contributors * small_contribs;
6780 int j, smallest, stop, start_n0;
6781 stbir__contributors * split_contribs = contribs + cur;
6782
6783 // scan for a max of 3x the filter width or until the next thread split
6784 stop = vertical_pixel_margin * 3;
6785 if ( each < stop )
6786 stop = each;
6787
6788 // loops a few times before early out
6789 smallest = 0;
6790 small_contribs = split_contribs;
6791 start_n0 = small_contribs->n0;
6792 for( j = 1 ; j <= stop ; j++ )
6793 {
6794 ++split_contribs;
6795 if ( split_contribs->n0 > start_n0 )
6796 break;
6797 if ( split_contribs->n0 < small_contribs->n0 )
6798 {
6799 small_contribs = split_contribs;
6800 smallest = j;
6801 }
6802 }
6803
6804 split_info[i-1].end_output_y += smallest;
6805 split_info[i].start_output_y += smallest;
6806 }
6807
6808 cur += each;
6809 left -= each;
6810
6811 // scatter range (updated to minimum as you run it)
6812 split_info[i].start_input_y = -vertical_pixel_margin;
6813 split_info[i].end_input_y = input_full_height + vertical_pixel_margin;
6814 }
6815}
6816
6817static void stbir__free_internal_mem( stbir__info *info )
6818{
6819 #define STBIR__FREE_AND_CLEAR( ptr ) { if ( ptr ) { void * p = (ptr); (ptr) = 0; STBIR_FREE( p, info->user_data); } }
6820
6821 if ( info )
6822 {
6823 #ifndef STBIR__SEPARATE_ALLOCATIONS
6824 STBIR__FREE_AND_CLEAR( info->alloced_mem );
6825 #else
6826 int i,j;
6827
6828 if ( ( info->vertical.gather_prescatter_contributors ) && ( (void*)info->vertical.gather_prescatter_contributors != (void*)info->split_info[0].decode_buffer ) )
6829 {
6830 STBIR__FREE_AND_CLEAR( info->vertical.gather_prescatter_coefficients );
6831 STBIR__FREE_AND_CLEAR( info->vertical.gather_prescatter_contributors );
6832 }
6833 for( i = 0 ; i < info->splits ; i++ )
6834 {
6835 for( j = 0 ; j < info->alloc_ring_buffer_num_entries ; j++ )
6836 {
6837 #ifdef STBIR_SIMD8
6838 if ( info->effective_channels == 3 )
6839 --info->split_info[i].ring_buffers[j]; // avx in 3 channel mode needs one float at the start of the buffer
6840 #endif
6841 STBIR__FREE_AND_CLEAR( info->split_info[i].ring_buffers[j] );
6842 }
6843
6844 #ifdef STBIR_SIMD8
6845 if ( info->effective_channels == 3 )
6846 --info->split_info[i].decode_buffer; // avx in 3 channel mode needs one float at the start of the buffer
6847 #endif
6848 STBIR__FREE_AND_CLEAR( info->split_info[i].decode_buffer );
6849 STBIR__FREE_AND_CLEAR( info->split_info[i].ring_buffers );
6850 STBIR__FREE_AND_CLEAR( info->split_info[i].vertical_buffer );
6851 }
6852 STBIR__FREE_AND_CLEAR( info->split_info );
6853 if ( info->vertical.coefficients != info->horizontal.coefficients )
6854 {
6855 STBIR__FREE_AND_CLEAR( info->vertical.coefficients );
6856 STBIR__FREE_AND_CLEAR( info->vertical.contributors );
6857 }
6858 STBIR__FREE_AND_CLEAR( info->horizontal.coefficients );
6859 STBIR__FREE_AND_CLEAR( info->horizontal.contributors );
6860 STBIR__FREE_AND_CLEAR( info->alloced_mem );
6861 STBIR_FREE( info, info->user_data );
6862 #endif
6863 }
6864
6865 #undef STBIR__FREE_AND_CLEAR
6866}
6867
6868static int stbir__get_max_split( int splits, int height )
6869{
6870 int i;
6871 int max = 0;
6872
6873 for( i = 0 ; i < splits ; i++ )
6874 {
6875 int each = height / ( splits - i );
6876 if ( each > max )
6877 max = each;
6878 height -= each;
6879 }
6880 return max;
6881}
6882
6883static stbir__horizontal_gather_channels_func ** stbir__horizontal_gather_n_coeffs_funcs[8] =
6884{
6885 0, stbir__horizontal_gather_1_channels_with_n_coeffs_funcs, stbir__horizontal_gather_2_channels_with_n_coeffs_funcs, stbir__horizontal_gather_3_channels_with_n_coeffs_funcs, stbir__horizontal_gather_4_channels_with_n_coeffs_funcs, 0,0, stbir__horizontal_gather_7_channels_with_n_coeffs_funcs
6886};
6887
6888static stbir__horizontal_gather_channels_func ** stbir__horizontal_gather_channels_funcs[8] =
6889{
6890 0, stbir__horizontal_gather_1_channels_funcs, stbir__horizontal_gather_2_channels_funcs, stbir__horizontal_gather_3_channels_funcs, stbir__horizontal_gather_4_channels_funcs, 0,0, stbir__horizontal_gather_7_channels_funcs
6891};
6892
6893// there are six resize classifications: 0 == vertical scatter, 1 == vertical gather < 1x scale, 2 == vertical gather 1x-2x scale, 4 == vertical gather < 3x scale, 4 == vertical gather > 3x scale, 5 == <=4 pixel height, 6 == <=4 pixel wide column
6894#define STBIR_RESIZE_CLASSIFICATIONS 8
6895
6896static float stbir__compute_weights[5][STBIR_RESIZE_CLASSIFICATIONS][4]= // 5 = 0=1chan, 1=2chan, 2=3chan, 3=4chan, 4=7chan
6897{
6898 {
6899 { 1.00000f, 1.00000f, 0.31250f, 1.00000f },
6900 { 0.56250f, 0.59375f, 0.00000f, 0.96875f },
6901 { 1.00000f, 0.06250f, 0.00000f, 1.00000f },
6902 { 0.00000f, 0.09375f, 1.00000f, 1.00000f },
6903 { 1.00000f, 1.00000f, 0.31250f, 1.00000f },
6904 { 0.03125f, 0.12500f, 1.00000f, 1.00000f },
6905 { 1.00000f, 1.00000f, 0.06250f, 1.00000f },
6906 { 0.00000f, 1.00000f, 0.00000f, 0.03125f },
6907 }, {
6908 { 0.00000f, 0.84375f, 0.00000f, 0.03125f },
6909 { 0.09375f, 0.93750f, 0.00000f, 0.78125f },
6910 { 0.87500f, 0.21875f, 0.00000f, 0.96875f },
6911 { 0.09375f, 0.09375f, 1.00000f, 1.00000f },
6912 { 0.00000f, 0.84375f, 0.00000f, 0.03125f },
6913 { 0.03125f, 0.12500f, 1.00000f, 1.00000f },
6914 { 1.00000f, 1.00000f, 0.06250f, 1.00000f },
6915 { 0.00000f, 1.00000f, 0.00000f, 0.53125f },
6916 }, {
6917 { 0.00000f, 0.53125f, 0.00000f, 0.03125f },
6918 { 0.06250f, 0.96875f, 0.00000f, 0.53125f },
6919 { 0.87500f, 0.18750f, 0.00000f, 0.93750f },
6920 { 0.00000f, 0.09375f, 1.00000f, 1.00000f },
6921 { 0.00000f, 0.53125f, 0.00000f, 0.03125f },
6922 { 0.03125f, 0.12500f, 1.00000f, 1.00000f },
6923 { 1.00000f, 1.00000f, 0.06250f, 1.00000f },
6924 { 0.00000f, 1.00000f, 0.00000f, 0.56250f },
6925 }, {
6926 { 0.00000f, 0.50000f, 0.00000f, 0.71875f },
6927 { 0.06250f, 0.84375f, 0.00000f, 0.87500f },
6928 { 1.00000f, 0.50000f, 0.50000f, 0.96875f },
6929 { 1.00000f, 0.09375f, 0.31250f, 0.50000f },
6930 { 0.00000f, 0.50000f, 0.00000f, 0.71875f },
6931 { 1.00000f, 0.03125f, 0.03125f, 0.53125f },
6932 { 1.00000f, 1.00000f, 0.06250f, 1.00000f },
6933 { 0.00000f, 1.00000f, 0.03125f, 0.18750f },
6934 }, {
6935 { 0.00000f, 0.59375f, 0.00000f, 0.96875f },
6936 { 0.06250f, 0.81250f, 0.06250f, 0.59375f },
6937 { 0.75000f, 0.43750f, 0.12500f, 0.96875f },
6938 { 0.87500f, 0.06250f, 0.18750f, 0.43750f },
6939 { 0.00000f, 0.59375f, 0.00000f, 0.96875f },
6940 { 0.15625f, 0.12500f, 1.00000f, 1.00000f },
6941 { 1.00000f, 1.00000f, 0.06250f, 1.00000f },
6942 { 0.00000f, 1.00000f, 0.03125f, 0.34375f },
6943 }
6944};
6945
6946// structure that allow us to query and override info for training the costs
6947typedef struct STBIR__V_FIRST_INFO
6948{
6949 double v_cost, h_cost;
6950 int control_v_first; // 0 = no control, 1 = force hori, 2 = force vert
6951 int v_first;
6952 int v_resize_classification;
6953 int is_gather;
6954} STBIR__V_FIRST_INFO;
6955
6956#ifdef STBIR__V_FIRST_INFO_BUFFER
6957static STBIR__V_FIRST_INFO STBIR__V_FIRST_INFO_BUFFER = {0};
6958#define STBIR__V_FIRST_INFO_POINTER &STBIR__V_FIRST_INFO_BUFFER
6959#else
6960#define STBIR__V_FIRST_INFO_POINTER 0
6961#endif
6962
6963// Figure out whether to scale along the horizontal or vertical first.
6964// This only *super* important when you are scaling by a massively
6965// different amount in the vertical vs the horizontal (for example, if
6966// you are scaling by 2x in the width, and 0.5x in the height, then you
6967// want to do the vertical scale first, because it's around 3x faster
6968// in that order.
6969//
6970// In more normal circumstances, this makes a 20-40% differences, so
6971// it's good to get right, but not critical. The normal way that you
6972// decide which direction goes first is just figuring out which
6973// direction does more multiplies. But with modern CPUs with their
6974// fancy caches and SIMD and high IPC abilities, so there's just a lot
6975// more that goes into it.
6976//
6977// My handwavy sort of solution is to have an app that does a whole
6978// bunch of timing for both vertical and horizontal first modes,
6979// and then another app that can read lots of these timing files
6980// and try to search for the best weights to use. Dotimings.c
6981// is the app that does a bunch of timings, and vf_train.c is the
6982// app that solves for the best weights (and shows how well it
6983// does currently).
6984
6985static int stbir__should_do_vertical_first( float weights_table[STBIR_RESIZE_CLASSIFICATIONS][4], int horizontal_filter_pixel_width, float horizontal_scale, int horizontal_output_size, int vertical_filter_pixel_width, float vertical_scale, int vertical_output_size, int is_gather, STBIR__V_FIRST_INFO * info )
6986{
6987 double v_cost, h_cost;
6988 float * weights;
6989 int vertical_first;
6990 int v_classification;
6991
6992 // categorize the resize into buckets
6993 if ( ( vertical_output_size <= 4 ) || ( horizontal_output_size <= 4 ) )
6994 v_classification = ( vertical_output_size < horizontal_output_size ) ? 6 : 7;
6995 else if ( ( !is_gather ) && ( ( vertical_output_size <= 16 ) || ( horizontal_output_size <= 16 ) ) )
6996 v_classification = 4;
6997 else if ( vertical_scale <= 1.0f )
6998 v_classification = ( is_gather ) ? 1 : 0;
6999 else if ( vertical_scale <= 2.0f)
7000 v_classification = 2;
7001 else if ( vertical_scale <= 3.0f)
7002 v_classification = 3;
7003 else
7004 v_classification = 5; // everything bigger than 3x
7005
7006 // use the right weights
7007 weights = weights_table[ v_classification ];
7008
7009 // this is the costs when you don't take into account modern CPUs with high ipc and simd and caches - wish we had a better estimate
7010 h_cost = (float)horizontal_filter_pixel_width * weights[0] + horizontal_scale * (float)vertical_filter_pixel_width * weights[1];
7011 v_cost = (float)vertical_filter_pixel_width * weights[2] + vertical_scale * (float)horizontal_filter_pixel_width * weights[3];
7012
7013 // use computation estimate to decide vertical first or not
7014 vertical_first = ( v_cost <= h_cost ) ? 1 : 0;
7015
7016 // save these, if requested
7017 if ( info )
7018 {
7019 info->h_cost = h_cost;
7020 info->v_cost = v_cost;
7021 info->v_resize_classification = v_classification;
7022 info->v_first = vertical_first;
7023 info->is_gather = is_gather;
7024 }
7025
7026 // and this allows us to override everything for testing (see dotiming.c)
7027 if ( ( info ) && ( info->control_v_first ) )
7028 vertical_first = ( info->control_v_first == 2 ) ? 1 : 0;
7029
7030 return vertical_first;
7031}
7032
7033// layout lookups - must match stbir_internal_pixel_layout
7034static unsigned char stbir__pixel_channels[] = {
7035 1,2,3,3,4, // 1ch, 2ch, rgb, bgr, 4ch
7036 4,4,4,4,2,2, // RGBA,BGRA,ARGB,ABGR,RA,AR
7037 4,4,4,4,2,2, // RGBA_PM,BGRA_PM,ARGB_PM,ABGR_PM,RA_PM,AR_PM
7038};
7039
7040// the internal pixel layout enums are in a different order, so we can easily do range comparisons of types
7041// the public pixel layout is ordered in a way that if you cast num_channels (1-4) to the enum, you get something sensible
7042static stbir_internal_pixel_layout stbir__pixel_layout_convert_public_to_internal[] = {
7043 STBIRI_BGR, STBIRI_1CHANNEL, STBIRI_2CHANNEL, STBIRI_RGB, STBIRI_RGBA,
7044 STBIRI_4CHANNEL, STBIRI_BGRA, STBIRI_ARGB, STBIRI_ABGR, STBIRI_RA, STBIRI_AR,
7045 STBIRI_RGBA_PM, STBIRI_BGRA_PM, STBIRI_ARGB_PM, STBIRI_ABGR_PM, STBIRI_RA_PM, STBIRI_AR_PM,
7046};
7047
7048static stbir__info * stbir__alloc_internal_mem_and_build_samplers( stbir__sampler * horizontal, stbir__sampler * vertical, stbir__contributors * conservative, stbir_pixel_layout input_pixel_layout_public, stbir_pixel_layout output_pixel_layout_public, int splits, int new_x, int new_y, int fast_alpha, void * user_data STBIR_ONLY_PROFILE_BUILD_GET_INFO )
7049{
7050 static char stbir_channel_count_index[8]={ 9,0,1,2, 3,9,9,4 };
7051
7052 stbir__info * info = 0;
7053 void * alloced = 0;
7054 size_t alloced_total = 0;
7055 int vertical_first;
7056 size_t decode_buffer_size, ring_buffer_length_bytes, ring_buffer_size, vertical_buffer_size;
7057 int alloc_ring_buffer_num_entries;
7058
7059 int alpha_weighting_type = 0; // 0=none, 1=simple, 2=fancy
7060 int conservative_split_output_size = stbir__get_max_split( splits, vertical->scale_info.output_sub_size );
7061 stbir_internal_pixel_layout input_pixel_layout = stbir__pixel_layout_convert_public_to_internal[ input_pixel_layout_public ];
7062 stbir_internal_pixel_layout output_pixel_layout = stbir__pixel_layout_convert_public_to_internal[ output_pixel_layout_public ];
7063 int channels = stbir__pixel_channels[ input_pixel_layout ];
7064 int effective_channels = channels;
7065
7066 // first figure out what type of alpha weighting to use (if any)
7067 if ( ( horizontal->filter_enum != STBIR_FILTER_POINT_SAMPLE ) || ( vertical->filter_enum != STBIR_FILTER_POINT_SAMPLE ) ) // no alpha weighting on point sampling
7068 {
7069 if ( ( input_pixel_layout >= STBIRI_RGBA ) && ( input_pixel_layout <= STBIRI_AR ) && ( output_pixel_layout >= STBIRI_RGBA ) && ( output_pixel_layout <= STBIRI_AR ) )
7070 {
7071 if ( fast_alpha )
7072 {
7073 alpha_weighting_type = 4;
7074 }
7075 else
7076 {
7077 static int fancy_alpha_effective_cnts[6] = { 7, 7, 7, 7, 3, 3 };
7078 alpha_weighting_type = 2;
7079 effective_channels = fancy_alpha_effective_cnts[ input_pixel_layout - STBIRI_RGBA ];
7080 }
7081 }
7082 else if ( ( input_pixel_layout >= STBIRI_RGBA_PM ) && ( input_pixel_layout <= STBIRI_AR_PM ) && ( output_pixel_layout >= STBIRI_RGBA ) && ( output_pixel_layout <= STBIRI_AR ) )
7083 {
7084 // input premult, output non-premult
7085 alpha_weighting_type = 3;
7086 }
7087 else if ( ( input_pixel_layout >= STBIRI_RGBA ) && ( input_pixel_layout <= STBIRI_AR ) && ( output_pixel_layout >= STBIRI_RGBA_PM ) && ( output_pixel_layout <= STBIRI_AR_PM ) )
7088 {
7089 // input non-premult, output premult
7090 alpha_weighting_type = 1;
7091 }
7092 }
7093
7094 // channel in and out count must match currently
7095 if ( channels != stbir__pixel_channels[ output_pixel_layout ] )
7096 return 0;
7097
7098 // get vertical first
7099 vertical_first = stbir__should_do_vertical_first( stbir__compute_weights[ (int)stbir_channel_count_index[ effective_channels ] ], horizontal->filter_pixel_width, horizontal->scale_info.scale, horizontal->scale_info.output_sub_size, vertical->filter_pixel_width, vertical->scale_info.scale, vertical->scale_info.output_sub_size, vertical->is_gather, STBIR__V_FIRST_INFO_POINTER );
7100
7101 // sometimes read one float off in some of the unrolled loops (with a weight of zero coeff, so it doesn't have an effect)
7102 // we use a few extra floats instead of just 1, so that input callback buffer can overlap with the decode buffer without
7103 // the conversion routines overwriting the callback input data.
7104 decode_buffer_size = ( conservative->n1 - conservative->n0 + 1 ) * effective_channels * sizeof(float) + sizeof(float)*STBIR_INPUT_CALLBACK_PADDING; // extra floats for input callback stagger
7105
7106#if defined( STBIR__SEPARATE_ALLOCATIONS ) && defined(STBIR_SIMD8)
7107 if ( effective_channels == 3 )
7108 decode_buffer_size += sizeof(float); // avx in 3 channel mode needs one float at the start of the buffer (only with separate allocations)
7109#endif
7110
7111 ring_buffer_length_bytes = (size_t)horizontal->scale_info.output_sub_size * (size_t)effective_channels * sizeof(float) + sizeof(float)*STBIR_INPUT_CALLBACK_PADDING; // extra floats for padding
7112
7113 // if we do vertical first, the ring buffer holds a whole decoded line
7114 if ( vertical_first )
7115 ring_buffer_length_bytes = ( decode_buffer_size + 15 ) & ~15;
7116
7117 if ( ( ring_buffer_length_bytes & 4095 ) == 0 ) ring_buffer_length_bytes += 64*3; // avoid 4k alias
7118
7119 // One extra entry because floating point precision problems sometimes cause an extra to be necessary.
7120 alloc_ring_buffer_num_entries = vertical->filter_pixel_width + 1;
7121
7122 // we never need more ring buffer entries than the scanlines we're outputting when in scatter mode
7123 if ( ( !vertical->is_gather ) && ( alloc_ring_buffer_num_entries > conservative_split_output_size ) )
7124 alloc_ring_buffer_num_entries = conservative_split_output_size;
7125
7126 ring_buffer_size = (size_t)alloc_ring_buffer_num_entries * (size_t)ring_buffer_length_bytes;
7127
7128 // The vertical buffer is used differently, depending on whether we are scattering
7129 // the vertical scanlines, or gathering them.
7130 // If scattering, it's used at the temp buffer to accumulate each output.
7131 // If gathering, it's just the output buffer.
7132 vertical_buffer_size = (size_t)horizontal->scale_info.output_sub_size * (size_t)effective_channels * sizeof(float) + sizeof(float); // extra float for padding
7133
7134 // we make two passes through this loop, 1st to add everything up, 2nd to allocate and init
7135 for(;;)
7136 {
7137 int i;
7138 void * advance_mem = alloced;
7139 int copy_horizontal = 0;
7140 stbir__sampler * possibly_use_horizontal_for_pivot = 0;
7141
7142#ifdef STBIR__SEPARATE_ALLOCATIONS
7143 #define STBIR__NEXT_PTR( ptr, size, ntype ) if ( alloced ) { void * p = STBIR_MALLOC( size, user_data); if ( p == 0 ) { stbir__free_internal_mem( info ); return 0; } (ptr) = (ntype*)p; }
7144#else
7145 #define STBIR__NEXT_PTR( ptr, size, ntype ) advance_mem = (void*) ( ( ((size_t)advance_mem) + 15 ) & ~15 ); if ( alloced ) ptr = (ntype*)advance_mem; advance_mem = (char*)(((size_t)advance_mem) + (size));
7146#endif
7147
7148 STBIR__NEXT_PTR( info, sizeof( stbir__info ), stbir__info );
7149
7150 STBIR__NEXT_PTR( info->split_info, sizeof( stbir__per_split_info ) * splits, stbir__per_split_info );
7151
7152 if ( info )
7153 {
7154 static stbir__alpha_weight_func * fancy_alpha_weights[6] = { stbir__fancy_alpha_weight_4ch, stbir__fancy_alpha_weight_4ch, stbir__fancy_alpha_weight_4ch, stbir__fancy_alpha_weight_4ch, stbir__fancy_alpha_weight_2ch, stbir__fancy_alpha_weight_2ch };
7155 static stbir__alpha_unweight_func * fancy_alpha_unweights[6] = { stbir__fancy_alpha_unweight_4ch, stbir__fancy_alpha_unweight_4ch, stbir__fancy_alpha_unweight_4ch, stbir__fancy_alpha_unweight_4ch, stbir__fancy_alpha_unweight_2ch, stbir__fancy_alpha_unweight_2ch };
7156 static stbir__alpha_weight_func * simple_alpha_weights[6] = { stbir__simple_alpha_weight_4ch, stbir__simple_alpha_weight_4ch, stbir__simple_alpha_weight_4ch, stbir__simple_alpha_weight_4ch, stbir__simple_alpha_weight_2ch, stbir__simple_alpha_weight_2ch };
7157 static stbir__alpha_unweight_func * simple_alpha_unweights[6] = { stbir__simple_alpha_unweight_4ch, stbir__simple_alpha_unweight_4ch, stbir__simple_alpha_unweight_4ch, stbir__simple_alpha_unweight_4ch, stbir__simple_alpha_unweight_2ch, stbir__simple_alpha_unweight_2ch };
7158
7159 // initialize info fields
7160 info->alloced_mem = alloced;
7161 info->alloced_total = alloced_total;
7162
7163 info->channels = channels;
7164 info->effective_channels = effective_channels;
7165
7166 info->offset_x = new_x;
7167 info->offset_y = new_y;
7168 info->alloc_ring_buffer_num_entries = (int)alloc_ring_buffer_num_entries;
7169 info->ring_buffer_num_entries = 0;
7170 info->ring_buffer_length_bytes = (int)ring_buffer_length_bytes;
7171 info->splits = splits;
7172 info->vertical_first = vertical_first;
7173
7174 info->input_pixel_layout_internal = input_pixel_layout;
7175 info->output_pixel_layout_internal = output_pixel_layout;
7176
7177 // setup alpha weight functions
7178 info->alpha_weight = 0;
7179 info->alpha_unweight = 0;
7180
7181 // handle alpha weighting functions and overrides
7182 if ( alpha_weighting_type == 2 )
7183 {
7184 // high quality alpha multiplying on the way in, dividing on the way out
7185 info->alpha_weight = fancy_alpha_weights[ input_pixel_layout - STBIRI_RGBA ];
7186 info->alpha_unweight = fancy_alpha_unweights[ output_pixel_layout - STBIRI_RGBA ];
7187 }
7188 else if ( alpha_weighting_type == 4 )
7189 {
7190 // fast alpha multiplying on the way in, dividing on the way out
7191 info->alpha_weight = simple_alpha_weights[ input_pixel_layout - STBIRI_RGBA ];
7192 info->alpha_unweight = simple_alpha_unweights[ output_pixel_layout - STBIRI_RGBA ];
7193 }
7194 else if ( alpha_weighting_type == 1 )
7195 {
7196 // fast alpha on the way in, leave in premultiplied form on way out
7197 info->alpha_weight = simple_alpha_weights[ input_pixel_layout - STBIRI_RGBA ];
7198 }
7199 else if ( alpha_weighting_type == 3 )
7200 {
7201 // incoming is premultiplied, fast alpha dividing on the way out - non-premultiplied output
7202 info->alpha_unweight = simple_alpha_unweights[ output_pixel_layout - STBIRI_RGBA ];
7203 }
7204
7205 // handle 3-chan color flipping, using the alpha weight path
7206 if ( ( ( input_pixel_layout == STBIRI_RGB ) && ( output_pixel_layout == STBIRI_BGR ) ) ||
7207 ( ( input_pixel_layout == STBIRI_BGR ) && ( output_pixel_layout == STBIRI_RGB ) ) )
7208 {
7209 // do the flipping on the smaller of the two ends
7210 if ( horizontal->scale_info.scale < 1.0f )
7211 info->alpha_unweight = stbir__simple_flip_3ch;
7212 else
7213 info->alpha_weight = stbir__simple_flip_3ch;
7214 }
7215
7216 }
7217
7218 // get all the per-split buffers
7219 for( i = 0 ; i < splits ; i++ )
7220 {
7221 STBIR__NEXT_PTR( info->split_info[i].decode_buffer, decode_buffer_size, float );
7222
7223#ifdef STBIR__SEPARATE_ALLOCATIONS
7224
7225 #ifdef STBIR_SIMD8
7226 if ( ( info ) && ( effective_channels == 3 ) )
7227 ++info->split_info[i].decode_buffer; // avx in 3 channel mode needs one float at the start of the buffer
7228 #endif
7229
7230 STBIR__NEXT_PTR( info->split_info[i].ring_buffers, alloc_ring_buffer_num_entries * sizeof(float*), float* );
7231 {
7232 int j;
7233 for( j = 0 ; j < alloc_ring_buffer_num_entries ; j++ )
7234 {
7235 STBIR__NEXT_PTR( info->split_info[i].ring_buffers[j], ring_buffer_length_bytes, float );
7236 #ifdef STBIR_SIMD8
7237 if ( ( info ) && ( effective_channels == 3 ) )
7238 ++info->split_info[i].ring_buffers[j]; // avx in 3 channel mode needs one float at the start of the buffer
7239 #endif
7240 }
7241 }
7242#else
7243 STBIR__NEXT_PTR( info->split_info[i].ring_buffer, ring_buffer_size, float );
7244#endif
7245 STBIR__NEXT_PTR( info->split_info[i].vertical_buffer, vertical_buffer_size, float );
7246 }
7247
7248 // alloc memory for to-be-pivoted coeffs (if necessary)
7249 if ( vertical->is_gather == 0 )
7250 {
7251 size_t both;
7252 size_t temp_mem_amt;
7253
7254 // when in vertical scatter mode, we first build the coefficients in gather mode, and then pivot after,
7255 // that means we need two buffers, so we try to use the decode buffer and ring buffer for this. if that
7256 // is too small, we just allocate extra memory to use as this temp.
7257
7258 both = (size_t)vertical->gather_prescatter_contributors_size + (size_t)vertical->gather_prescatter_coefficients_size;
7259
7260#ifdef STBIR__SEPARATE_ALLOCATIONS
7261 temp_mem_amt = decode_buffer_size;
7262
7263 #ifdef STBIR_SIMD8
7264 if ( effective_channels == 3 )
7265 --temp_mem_amt; // avx in 3 channel mode needs one float at the start of the buffer
7266 #endif
7267#else
7268 temp_mem_amt = (size_t)( decode_buffer_size + ring_buffer_size + vertical_buffer_size ) * (size_t)splits;
7269#endif
7270 if ( temp_mem_amt >= both )
7271 {
7272 if ( info )
7273 {
7274 vertical->gather_prescatter_contributors = (stbir__contributors*)info->split_info[0].decode_buffer;
7275 vertical->gather_prescatter_coefficients = (float*) ( ( (char*)info->split_info[0].decode_buffer ) + vertical->gather_prescatter_contributors_size );
7276 }
7277 }
7278 else
7279 {
7280 // ring+decode memory is too small, so allocate temp memory
7281 STBIR__NEXT_PTR( vertical->gather_prescatter_contributors, vertical->gather_prescatter_contributors_size, stbir__contributors );
7282 STBIR__NEXT_PTR( vertical->gather_prescatter_coefficients, vertical->gather_prescatter_coefficients_size, float );
7283 }
7284 }
7285
7286 STBIR__NEXT_PTR( horizontal->contributors, horizontal->contributors_size, stbir__contributors );
7287 STBIR__NEXT_PTR( horizontal->coefficients, horizontal->coefficients_size, float );
7288
7289 // are the two filters identical?? (happens a lot with mipmap generation)
7290 if ( ( horizontal->filter_kernel == vertical->filter_kernel ) && ( horizontal->filter_support == vertical->filter_support ) && ( horizontal->edge == vertical->edge ) && ( horizontal->scale_info.output_sub_size == vertical->scale_info.output_sub_size ) )
7291 {
7292 float diff_scale = horizontal->scale_info.scale - vertical->scale_info.scale;
7293 float diff_shift = horizontal->scale_info.pixel_shift - vertical->scale_info.pixel_shift;
7294 if ( diff_scale < 0.0f ) diff_scale = -diff_scale;
7295 if ( diff_shift < 0.0f ) diff_shift = -diff_shift;
7296 if ( ( diff_scale <= stbir__small_float ) && ( diff_shift <= stbir__small_float ) )
7297 {
7298 if ( horizontal->is_gather == vertical->is_gather )
7299 {
7300 copy_horizontal = 1;
7301 goto no_vert_alloc;
7302 }
7303 // everything matches, but vertical is scatter, horizontal is gather, use horizontal coeffs for vertical pivot coeffs
7304 possibly_use_horizontal_for_pivot = horizontal;
7305 }
7306 }
7307
7308 STBIR__NEXT_PTR( vertical->contributors, vertical->contributors_size, stbir__contributors );
7309 STBIR__NEXT_PTR( vertical->coefficients, vertical->coefficients_size, float );
7310
7311 no_vert_alloc:
7312
7313 if ( info )
7314 {
7315 STBIR_PROFILE_BUILD_START( horizontal );
7316
7317 stbir__calculate_filters( horizontal, 0, user_data STBIR_ONLY_PROFILE_BUILD_SET_INFO );
7318
7319 // setup the horizontal gather functions
7320 // start with defaulting to the n_coeffs functions (specialized on channels and remnant leftover)
7321 info->horizontal_gather_channels = stbir__horizontal_gather_n_coeffs_funcs[ effective_channels ][ horizontal->extent_info.widest & 3 ];
7322 // but if the number of coeffs <= 12, use another set of special cases. <=12 coeffs is any enlarging resize, or shrinking resize down to about 1/3 size
7323 if ( horizontal->extent_info.widest <= 12 )
7324 info->horizontal_gather_channels = stbir__horizontal_gather_channels_funcs[ effective_channels ][ horizontal->extent_info.widest - 1 ];
7325
7326 info->scanline_extents.conservative.n0 = conservative->n0;
7327 info->scanline_extents.conservative.n1 = conservative->n1;
7328
7329 // get exact extents
7330 stbir__get_extents( horizontal, &info->scanline_extents );
7331
7332 // pack the horizontal coeffs
7333 horizontal->coefficient_width = stbir__pack_coefficients(horizontal->num_contributors, horizontal->contributors, horizontal->coefficients, horizontal->coefficient_width, horizontal->extent_info.widest, info->scanline_extents.conservative.n0, info->scanline_extents.conservative.n1 );
7334
7335 STBIR_MEMCPY( &info->horizontal, horizontal, sizeof( stbir__sampler ) );
7336
7337 STBIR_PROFILE_BUILD_END( horizontal );
7338
7339 if ( copy_horizontal )
7340 {
7341 STBIR_MEMCPY( &info->vertical, horizontal, sizeof( stbir__sampler ) );
7342 }
7343 else
7344 {
7345 STBIR_PROFILE_BUILD_START( vertical );
7346
7347 stbir__calculate_filters( vertical, possibly_use_horizontal_for_pivot, user_data STBIR_ONLY_PROFILE_BUILD_SET_INFO );
7348 STBIR_MEMCPY( &info->vertical, vertical, sizeof( stbir__sampler ) );
7349
7350 STBIR_PROFILE_BUILD_END( vertical );
7351 }
7352
7353 // setup the vertical split ranges
7354 stbir__get_split_info( info->split_info, info->splits, info->vertical.scale_info.output_sub_size, info->vertical.filter_pixel_margin, info->vertical.scale_info.input_full_size, info->vertical.is_gather, info->vertical.contributors );
7355
7356 // now we know precisely how many entries we need
7357 info->ring_buffer_num_entries = info->vertical.extent_info.widest;
7358
7359 // we never need more ring buffer entries than the scanlines we're outputting
7360 if ( ( !info->vertical.is_gather ) && ( info->ring_buffer_num_entries > conservative_split_output_size ) )
7361 info->ring_buffer_num_entries = conservative_split_output_size;
7362 STBIR_ASSERT( info->ring_buffer_num_entries <= info->alloc_ring_buffer_num_entries );
7363 }
7364 #undef STBIR__NEXT_PTR
7365
7366
7367 // is this the first time through loop?
7368 if ( info == 0 )
7369 {
7370 alloced_total = ( 15 + (size_t)advance_mem );
7371 alloced = STBIR_MALLOC( alloced_total, user_data );
7372 if ( alloced == 0 )
7373 return 0;
7374 }
7375 else
7376 return info; // success
7377 }
7378}
7379
7380static int stbir__perform_resize( stbir__info const * info, int split_start, int split_count )
7381{
7382 stbir__per_split_info * split_info = info->split_info + split_start;
7383
7384 STBIR_PROFILE_CLEAR_EXTRAS();
7385
7386 STBIR_PROFILE_FIRST_START( looping );
7387 if (info->vertical.is_gather)
7388 stbir__vertical_gather_loop( info, split_info, split_count );
7389 else
7390 stbir__vertical_scatter_loop( info, split_info, split_count );
7391 STBIR_PROFILE_END( looping );
7392
7393 return 1;
7394}
7395
7396static void stbir__update_info_from_resize( stbir__info * info, STBIR_RESIZE * resize )
7397{
7398 static stbir__decode_pixels_func * decode_simple[STBIR_TYPE_HALF_FLOAT-STBIR_TYPE_UINT8_SRGB+1]=
7399 {
7400 /* 1ch-4ch */ stbir__decode_uint8_srgb, stbir__decode_uint8_srgb, 0, stbir__decode_float_linear, stbir__decode_half_float_linear,
7401 };
7402
7403 static stbir__decode_pixels_func * decode_alphas[STBIRI_AR-STBIRI_RGBA+1][STBIR_TYPE_HALF_FLOAT-STBIR_TYPE_UINT8_SRGB+1]=
7404 {
7405 { /* RGBA */ stbir__decode_uint8_srgb4_linearalpha, stbir__decode_uint8_srgb, 0, stbir__decode_float_linear, stbir__decode_half_float_linear },
7406 { /* BGRA */ stbir__decode_uint8_srgb4_linearalpha_BGRA, stbir__decode_uint8_srgb_BGRA, 0, stbir__decode_float_linear_BGRA, stbir__decode_half_float_linear_BGRA },
7407 { /* ARGB */ stbir__decode_uint8_srgb4_linearalpha_ARGB, stbir__decode_uint8_srgb_ARGB, 0, stbir__decode_float_linear_ARGB, stbir__decode_half_float_linear_ARGB },
7408 { /* ABGR */ stbir__decode_uint8_srgb4_linearalpha_ABGR, stbir__decode_uint8_srgb_ABGR, 0, stbir__decode_float_linear_ABGR, stbir__decode_half_float_linear_ABGR },
7409 { /* RA */ stbir__decode_uint8_srgb2_linearalpha, stbir__decode_uint8_srgb, 0, stbir__decode_float_linear, stbir__decode_half_float_linear },
7410 { /* AR */ stbir__decode_uint8_srgb2_linearalpha_AR, stbir__decode_uint8_srgb_AR, 0, stbir__decode_float_linear_AR, stbir__decode_half_float_linear_AR },
7411 };
7412
7413 static stbir__decode_pixels_func * decode_simple_scaled_or_not[2][2]=
7414 {
7415 { stbir__decode_uint8_linear_scaled, stbir__decode_uint8_linear }, { stbir__decode_uint16_linear_scaled, stbir__decode_uint16_linear },
7416 };
7417
7418 static stbir__decode_pixels_func * decode_alphas_scaled_or_not[STBIRI_AR-STBIRI_RGBA+1][2][2]=
7419 {
7420 { /* RGBA */ { stbir__decode_uint8_linear_scaled, stbir__decode_uint8_linear }, { stbir__decode_uint16_linear_scaled, stbir__decode_uint16_linear } },
7421 { /* BGRA */ { stbir__decode_uint8_linear_scaled_BGRA, stbir__decode_uint8_linear_BGRA }, { stbir__decode_uint16_linear_scaled_BGRA, stbir__decode_uint16_linear_BGRA } },
7422 { /* ARGB */ { stbir__decode_uint8_linear_scaled_ARGB, stbir__decode_uint8_linear_ARGB }, { stbir__decode_uint16_linear_scaled_ARGB, stbir__decode_uint16_linear_ARGB } },
7423 { /* ABGR */ { stbir__decode_uint8_linear_scaled_ABGR, stbir__decode_uint8_linear_ABGR }, { stbir__decode_uint16_linear_scaled_ABGR, stbir__decode_uint16_linear_ABGR } },
7424 { /* RA */ { stbir__decode_uint8_linear_scaled, stbir__decode_uint8_linear }, { stbir__decode_uint16_linear_scaled, stbir__decode_uint16_linear } },
7425 { /* AR */ { stbir__decode_uint8_linear_scaled_AR, stbir__decode_uint8_linear_AR }, { stbir__decode_uint16_linear_scaled_AR, stbir__decode_uint16_linear_AR } }
7426 };
7427
7428 static stbir__encode_pixels_func * encode_simple[STBIR_TYPE_HALF_FLOAT-STBIR_TYPE_UINT8_SRGB+1]=
7429 {
7430 /* 1ch-4ch */ stbir__encode_uint8_srgb, stbir__encode_uint8_srgb, 0, stbir__encode_float_linear, stbir__encode_half_float_linear,
7431 };
7432
7433 static stbir__encode_pixels_func * encode_alphas[STBIRI_AR-STBIRI_RGBA+1][STBIR_TYPE_HALF_FLOAT-STBIR_TYPE_UINT8_SRGB+1]=
7434 {
7435 { /* RGBA */ stbir__encode_uint8_srgb4_linearalpha, stbir__encode_uint8_srgb, 0, stbir__encode_float_linear, stbir__encode_half_float_linear },
7436 { /* BGRA */ stbir__encode_uint8_srgb4_linearalpha_BGRA, stbir__encode_uint8_srgb_BGRA, 0, stbir__encode_float_linear_BGRA, stbir__encode_half_float_linear_BGRA },
7437 { /* ARGB */ stbir__encode_uint8_srgb4_linearalpha_ARGB, stbir__encode_uint8_srgb_ARGB, 0, stbir__encode_float_linear_ARGB, stbir__encode_half_float_linear_ARGB },
7438 { /* ABGR */ stbir__encode_uint8_srgb4_linearalpha_ABGR, stbir__encode_uint8_srgb_ABGR, 0, stbir__encode_float_linear_ABGR, stbir__encode_half_float_linear_ABGR },
7439 { /* RA */ stbir__encode_uint8_srgb2_linearalpha, stbir__encode_uint8_srgb, 0, stbir__encode_float_linear, stbir__encode_half_float_linear },
7440 { /* AR */ stbir__encode_uint8_srgb2_linearalpha_AR, stbir__encode_uint8_srgb_AR, 0, stbir__encode_float_linear_AR, stbir__encode_half_float_linear_AR }
7441 };
7442
7443 static stbir__encode_pixels_func * encode_simple_scaled_or_not[2][2]=
7444 {
7445 { stbir__encode_uint8_linear_scaled, stbir__encode_uint8_linear }, { stbir__encode_uint16_linear_scaled, stbir__encode_uint16_linear },
7446 };
7447
7448 static stbir__encode_pixels_func * encode_alphas_scaled_or_not[STBIRI_AR-STBIRI_RGBA+1][2][2]=
7449 {
7450 { /* RGBA */ { stbir__encode_uint8_linear_scaled, stbir__encode_uint8_linear }, { stbir__encode_uint16_linear_scaled, stbir__encode_uint16_linear } },
7451 { /* BGRA */ { stbir__encode_uint8_linear_scaled_BGRA, stbir__encode_uint8_linear_BGRA }, { stbir__encode_uint16_linear_scaled_BGRA, stbir__encode_uint16_linear_BGRA } },
7452 { /* ARGB */ { stbir__encode_uint8_linear_scaled_ARGB, stbir__encode_uint8_linear_ARGB }, { stbir__encode_uint16_linear_scaled_ARGB, stbir__encode_uint16_linear_ARGB } },
7453 { /* ABGR */ { stbir__encode_uint8_linear_scaled_ABGR, stbir__encode_uint8_linear_ABGR }, { stbir__encode_uint16_linear_scaled_ABGR, stbir__encode_uint16_linear_ABGR } },
7454 { /* RA */ { stbir__encode_uint8_linear_scaled, stbir__encode_uint8_linear }, { stbir__encode_uint16_linear_scaled, stbir__encode_uint16_linear } },
7455 { /* AR */ { stbir__encode_uint8_linear_scaled_AR, stbir__encode_uint8_linear_AR }, { stbir__encode_uint16_linear_scaled_AR, stbir__encode_uint16_linear_AR } }
7456 };
7457
7458 stbir__decode_pixels_func * decode_pixels = 0;
7459 stbir__encode_pixels_func * encode_pixels = 0;
7460 stbir_datatype input_type, output_type;
7461
7462 input_type = resize->input_data_type;
7463 output_type = resize->output_data_type;
7464 info->input_data = resize->input_pixels;
7465 info->input_stride_bytes = resize->input_stride_in_bytes;
7466 info->output_stride_bytes = resize->output_stride_in_bytes;
7467
7468 // if we're completely point sampling, then we can turn off SRGB
7469 if ( ( info->horizontal.filter_enum == STBIR_FILTER_POINT_SAMPLE ) && ( info->vertical.filter_enum == STBIR_FILTER_POINT_SAMPLE ) )
7470 {
7471 if ( ( ( input_type == STBIR_TYPE_UINT8_SRGB ) || ( input_type == STBIR_TYPE_UINT8_SRGB_ALPHA ) ) &&
7472 ( ( output_type == STBIR_TYPE_UINT8_SRGB ) || ( output_type == STBIR_TYPE_UINT8_SRGB_ALPHA ) ) )
7473 {
7474 input_type = STBIR_TYPE_UINT8;
7475 output_type = STBIR_TYPE_UINT8;
7476 }
7477 }
7478
7479 // recalc the output and input strides
7480 if ( info->input_stride_bytes == 0 )
7481 info->input_stride_bytes = info->channels * info->horizontal.scale_info.input_full_size * stbir__type_size[input_type];
7482
7483 if ( info->output_stride_bytes == 0 )
7484 info->output_stride_bytes = info->channels * info->horizontal.scale_info.output_sub_size * stbir__type_size[output_type];
7485
7486 // calc offset
7487 info->output_data = ( (char*) resize->output_pixels ) + ( (size_t) info->offset_y * (size_t) resize->output_stride_in_bytes ) + ( info->offset_x * info->channels * stbir__type_size[output_type] );
7488
7489 info->in_pixels_cb = resize->input_cb;
7490 info->user_data = resize->user_data;
7491 info->out_pixels_cb = resize->output_cb;
7492
7493 // setup the input format converters
7494 if ( ( input_type == STBIR_TYPE_UINT8 ) || ( input_type == STBIR_TYPE_UINT16 ) )
7495 {
7496 int non_scaled = 0;
7497
7498 // check if we can run unscaled - 0-255.0/0-65535.0 instead of 0-1.0 (which is a tiny bit faster when doing linear 8->8 or 16->16)
7499 if ( ( !info->alpha_weight ) && ( !info->alpha_unweight ) ) // don't short circuit when alpha weighting (get everything to 0-1.0 as usual)
7500 if ( ( ( input_type == STBIR_TYPE_UINT8 ) && ( output_type == STBIR_TYPE_UINT8 ) ) || ( ( input_type == STBIR_TYPE_UINT16 ) && ( output_type == STBIR_TYPE_UINT16 ) ) )
7501 non_scaled = 1;
7502
7503 if ( info->input_pixel_layout_internal <= STBIRI_4CHANNEL )
7504 decode_pixels = decode_simple_scaled_or_not[ input_type == STBIR_TYPE_UINT16 ][ non_scaled ];
7505 else
7506 decode_pixels = decode_alphas_scaled_or_not[ ( info->input_pixel_layout_internal - STBIRI_RGBA ) % ( STBIRI_AR-STBIRI_RGBA+1 ) ][ input_type == STBIR_TYPE_UINT16 ][ non_scaled ];
7507 }
7508 else
7509 {
7510 if ( info->input_pixel_layout_internal <= STBIRI_4CHANNEL )
7511 decode_pixels = decode_simple[ input_type - STBIR_TYPE_UINT8_SRGB ];
7512 else
7513 decode_pixels = decode_alphas[ ( info->input_pixel_layout_internal - STBIRI_RGBA ) % ( STBIRI_AR-STBIRI_RGBA+1 ) ][ input_type - STBIR_TYPE_UINT8_SRGB ];
7514 }
7515
7516 // setup the output format converters
7517 if ( ( output_type == STBIR_TYPE_UINT8 ) || ( output_type == STBIR_TYPE_UINT16 ) )
7518 {
7519 int non_scaled = 0;
7520
7521 // check if we can run unscaled - 0-255.0/0-65535.0 instead of 0-1.0 (which is a tiny bit faster when doing linear 8->8 or 16->16)
7522 if ( ( !info->alpha_weight ) && ( !info->alpha_unweight ) ) // don't short circuit when alpha weighting (get everything to 0-1.0 as usual)
7523 if ( ( ( input_type == STBIR_TYPE_UINT8 ) && ( output_type == STBIR_TYPE_UINT8 ) ) || ( ( input_type == STBIR_TYPE_UINT16 ) && ( output_type == STBIR_TYPE_UINT16 ) ) )
7524 non_scaled = 1;
7525
7526 if ( info->output_pixel_layout_internal <= STBIRI_4CHANNEL )
7527 encode_pixels = encode_simple_scaled_or_not[ output_type == STBIR_TYPE_UINT16 ][ non_scaled ];
7528 else
7529 encode_pixels = encode_alphas_scaled_or_not[ ( info->output_pixel_layout_internal - STBIRI_RGBA ) % ( STBIRI_AR-STBIRI_RGBA+1 ) ][ output_type == STBIR_TYPE_UINT16 ][ non_scaled ];
7530 }
7531 else
7532 {
7533 if ( info->output_pixel_layout_internal <= STBIRI_4CHANNEL )
7534 encode_pixels = encode_simple[ output_type - STBIR_TYPE_UINT8_SRGB ];
7535 else
7536 encode_pixels = encode_alphas[ ( info->output_pixel_layout_internal - STBIRI_RGBA ) % ( STBIRI_AR-STBIRI_RGBA+1 ) ][ output_type - STBIR_TYPE_UINT8_SRGB ];
7537 }
7538
7539 info->input_type = input_type;
7540 info->output_type = output_type;
7541 info->decode_pixels = decode_pixels;
7542 info->encode_pixels = encode_pixels;
7543}
7544
7545static void stbir__clip( int * outx, int * outsubw, int outw, double * u0, double * u1 )
7546{
7547 double per, adj;
7548 int over;
7549
7550 // do left/top edge
7551 if ( *outx < 0 )
7552 {
7553 per = ( (double)*outx ) / ( (double)*outsubw ); // is negative
7554 adj = per * ( *u1 - *u0 );
7555 *u0 -= adj; // increases u0
7556 *outx = 0;
7557 }
7558
7559 // do right/bot edge
7560 over = outw - ( *outx + *outsubw );
7561 if ( over < 0 )
7562 {
7563 per = ( (double)over ) / ( (double)*outsubw ); // is negative
7564 adj = per * ( *u1 - *u0 );
7565 *u1 += adj; // decrease u1
7566 *outsubw = outw - *outx;
7567 }
7568}
7569
7570// converts a double to a rational that has less than one float bit of error (returns 0 if unable to do so)
7571static int stbir__double_to_rational(double f, stbir_uint32 limit, stbir_uint32 *numer, stbir_uint32 *denom, int limit_denom ) // limit_denom (1) or limit numer (0)
7572{
7573 double err;
7574 stbir_uint64 top, bot;
7575 stbir_uint64 numer_last = 0;
7576 stbir_uint64 denom_last = 1;
7577 stbir_uint64 numer_estimate = 1;
7578 stbir_uint64 denom_estimate = 0;
7579
7580 // scale to past float error range
7581 top = (stbir_uint64)( f * (double)(1 << 25) );
7582 bot = 1 << 25;
7583
7584 // keep refining, but usually stops in a few loops - usually 5 for bad cases
7585 for(;;)
7586 {
7587 stbir_uint64 est, temp;
7588
7589 // hit limit, break out and do best full range estimate
7590 if ( ( ( limit_denom ) ? denom_estimate : numer_estimate ) >= limit )
7591 break;
7592
7593 // is the current error less than 1 bit of a float? if so, we're done
7594 if ( denom_estimate )
7595 {
7596 err = ( (double)numer_estimate / (double)denom_estimate ) - f;
7597 if ( err < 0.0 ) err = -err;
7598 if ( err < ( 1.0 / (double)(1<<24) ) )
7599 {
7600 // yup, found it
7601 *numer = (stbir_uint32) numer_estimate;
7602 *denom = (stbir_uint32) denom_estimate;
7603 return 1;
7604 }
7605 }
7606
7607 // no more refinement bits left? break out and do full range estimate
7608 if ( bot == 0 )
7609 break;
7610
7611 // gcd the estimate bits
7612 est = top / bot;
7613 temp = top % bot;
7614 top = bot;
7615 bot = temp;
7616
7617 // move remainders
7618 temp = est * denom_estimate + denom_last;
7619 denom_last = denom_estimate;
7620 denom_estimate = temp;
7621
7622 // move remainders
7623 temp = est * numer_estimate + numer_last;
7624 numer_last = numer_estimate;
7625 numer_estimate = temp;
7626 }
7627
7628 // we didn't find anything good enough for float, use a full range estimate
7629 if ( limit_denom )
7630 {
7631 numer_estimate= (stbir_uint64)( f * (double)limit + 0.5 );
7632 denom_estimate = limit;
7633 }
7634 else
7635 {
7636 numer_estimate = limit;
7637 denom_estimate = (stbir_uint64)( ( (double)limit / f ) + 0.5 );
7638 }
7639
7640 *numer = (stbir_uint32) numer_estimate;
7641 *denom = (stbir_uint32) denom_estimate;
7642
7643 err = ( denom_estimate ) ? ( ( (double)(stbir_uint32)numer_estimate / (double)(stbir_uint32)denom_estimate ) - f ) : 1.0;
7644 if ( err < 0.0 ) err = -err;
7645 return ( err < ( 1.0 / (double)(1<<24) ) ) ? 1 : 0;
7646}
7647
7648static int stbir__calculate_region_transform( stbir__scale_info * scale_info, int output_full_range, int * output_offset, int output_sub_range, int input_full_range, double input_s0, double input_s1 )
7649{
7650 double output_range, input_range, output_s, input_s, ratio, scale;
7651
7652 input_s = input_s1 - input_s0;
7653
7654 // null area
7655 if ( ( output_full_range == 0 ) || ( input_full_range == 0 ) ||
7656 ( output_sub_range == 0 ) || ( input_s <= stbir__small_float ) )
7657 return 0;
7658
7659 // are either of the ranges completely out of bounds?
7660 if ( ( *output_offset >= output_full_range ) || ( ( *output_offset + output_sub_range ) <= 0 ) || ( input_s0 >= (1.0f-stbir__small_float) ) || ( input_s1 <= stbir__small_float ) )
7661 return 0;
7662
7663 output_range = (double)output_full_range;
7664 input_range = (double)input_full_range;
7665
7666 output_s = ( (double)output_sub_range) / output_range;
7667
7668 // figure out the scaling to use
7669 ratio = output_s / input_s;
7670
7671 // save scale before clipping
7672 scale = ( output_range / input_range ) * ratio;
7673 scale_info->scale = (float)scale;
7674 scale_info->inv_scale = (float)( 1.0 / scale );
7675
7676 // clip output area to left/right output edges (and adjust input area)
7677 stbir__clip( output_offset, &output_sub_range, output_full_range, &input_s0, &input_s1 );
7678
7679 // recalc input area
7680 input_s = input_s1 - input_s0;
7681
7682 // after clipping do we have zero input area?
7683 if ( input_s <= stbir__small_float )
7684 return 0;
7685
7686 // calculate and store the starting source offsets in output pixel space
7687 scale_info->pixel_shift = (float) ( input_s0 * ratio * output_range );
7688
7689 scale_info->scale_is_rational = stbir__double_to_rational( scale, ( scale <= 1.0 ) ? output_full_range : input_full_range, &scale_info->scale_numerator, &scale_info->scale_denominator, ( scale >= 1.0 ) );
7690
7691 scale_info->input_full_size = input_full_range;
7692 scale_info->output_sub_size = output_sub_range;
7693
7694 return 1;
7695}
7696
7697
7698static void stbir__init_and_set_layout( STBIR_RESIZE * resize, stbir_pixel_layout pixel_layout, stbir_datatype data_type )
7699{
7700 resize->input_cb = 0;
7701 resize->output_cb = 0;
7702 resize->user_data = resize;
7703 resize->samplers = 0;
7704 resize->called_alloc = 0;
7705 resize->horizontal_filter = STBIR_FILTER_DEFAULT;
7706 resize->horizontal_filter_kernel = 0; resize->horizontal_filter_support = 0;
7707 resize->vertical_filter = STBIR_FILTER_DEFAULT;
7708 resize->vertical_filter_kernel = 0; resize->vertical_filter_support = 0;
7709 resize->horizontal_edge = STBIR_EDGE_CLAMP;
7710 resize->vertical_edge = STBIR_EDGE_CLAMP;
7711 resize->input_s0 = 0; resize->input_t0 = 0; resize->input_s1 = 1; resize->input_t1 = 1;
7712 resize->output_subx = 0; resize->output_suby = 0; resize->output_subw = resize->output_w; resize->output_subh = resize->output_h;
7713 resize->input_data_type = data_type;
7714 resize->output_data_type = data_type;
7715 resize->input_pixel_layout_public = pixel_layout;
7716 resize->output_pixel_layout_public = pixel_layout;
7717 resize->needs_rebuild = 1;
7718}
7719
7720STBIRDEF void stbir_resize_init( STBIR_RESIZE * resize,
7721 const void *input_pixels, int input_w, int input_h, int input_stride_in_bytes, // stride can be zero
7722 void *output_pixels, int output_w, int output_h, int output_stride_in_bytes, // stride can be zero
7723 stbir_pixel_layout pixel_layout, stbir_datatype data_type )
7724{
7725 resize->input_pixels = input_pixels;
7726 resize->input_w = input_w;
7727 resize->input_h = input_h;
7728 resize->input_stride_in_bytes = input_stride_in_bytes;
7729 resize->output_pixels = output_pixels;
7730 resize->output_w = output_w;
7731 resize->output_h = output_h;
7732 resize->output_stride_in_bytes = output_stride_in_bytes;
7733 resize->fast_alpha = 0;
7734
7735 stbir__init_and_set_layout( resize, pixel_layout, data_type );
7736}
7737
7738// You can update parameters any time after resize_init
7739STBIRDEF void stbir_set_datatypes( STBIR_RESIZE * resize, stbir_datatype input_type, stbir_datatype output_type ) // by default, datatype from resize_init
7740{
7741 resize->input_data_type = input_type;
7742 resize->output_data_type = output_type;
7743 if ( ( resize->samplers ) && ( !resize->needs_rebuild ) )
7744 stbir__update_info_from_resize( resize->samplers, resize );
7745}
7746
7747STBIRDEF void stbir_set_pixel_callbacks( STBIR_RESIZE * resize, stbir_input_callback * input_cb, stbir_output_callback * output_cb ) // no callbacks by default
7748{
7749 resize->input_cb = input_cb;
7750 resize->output_cb = output_cb;
7751
7752 if ( ( resize->samplers ) && ( !resize->needs_rebuild ) )
7753 {
7754 resize->samplers->in_pixels_cb = input_cb;
7755 resize->samplers->out_pixels_cb = output_cb;
7756 }
7757}
7758
7759STBIRDEF void stbir_set_user_data( STBIR_RESIZE * resize, void * user_data ) // pass back STBIR_RESIZE* by default
7760{
7761 resize->user_data = user_data;
7762 if ( ( resize->samplers ) && ( !resize->needs_rebuild ) )
7763 resize->samplers->user_data = user_data;
7764}
7765
7766STBIRDEF void stbir_set_buffer_ptrs( STBIR_RESIZE * resize, const void * input_pixels, int input_stride_in_bytes, void * output_pixels, int output_stride_in_bytes )
7767{
7768 resize->input_pixels = input_pixels;
7769 resize->input_stride_in_bytes = input_stride_in_bytes;
7770 resize->output_pixels = output_pixels;
7771 resize->output_stride_in_bytes = output_stride_in_bytes;
7772 if ( ( resize->samplers ) && ( !resize->needs_rebuild ) )
7773 stbir__update_info_from_resize( resize->samplers, resize );
7774}
7775
7776
7777STBIRDEF int stbir_set_edgemodes( STBIR_RESIZE * resize, stbir_edge horizontal_edge, stbir_edge vertical_edge ) // CLAMP by default
7778{
7779 resize->horizontal_edge = horizontal_edge;
7780 resize->vertical_edge = vertical_edge;
7781 resize->needs_rebuild = 1;
7782 return 1;
7783}
7784
7785STBIRDEF int stbir_set_filters( STBIR_RESIZE * resize, stbir_filter horizontal_filter, stbir_filter vertical_filter ) // STBIR_DEFAULT_FILTER_UPSAMPLE/DOWNSAMPLE by default
7786{
7787 resize->horizontal_filter = horizontal_filter;
7788 resize->vertical_filter = vertical_filter;
7789 resize->needs_rebuild = 1;
7790 return 1;
7791}
7792
7793STBIRDEF int stbir_set_filter_callbacks( STBIR_RESIZE * resize, stbir__kernel_callback * horizontal_filter, stbir__support_callback * horizontal_support, stbir__kernel_callback * vertical_filter, stbir__support_callback * vertical_support )
7794{
7795 resize->horizontal_filter_kernel = horizontal_filter; resize->horizontal_filter_support = horizontal_support;
7796 resize->vertical_filter_kernel = vertical_filter; resize->vertical_filter_support = vertical_support;
7797 resize->needs_rebuild = 1;
7798 return 1;
7799}
7800
7801STBIRDEF int stbir_set_pixel_layouts( STBIR_RESIZE * resize, stbir_pixel_layout input_pixel_layout, stbir_pixel_layout output_pixel_layout ) // sets new pixel layouts
7802{
7803 resize->input_pixel_layout_public = input_pixel_layout;
7804 resize->output_pixel_layout_public = output_pixel_layout;
7805 resize->needs_rebuild = 1;
7806 return 1;
7807}
7808
7809
7810STBIRDEF int stbir_set_non_pm_alpha_speed_over_quality( STBIR_RESIZE * resize, int non_pma_alpha_speed_over_quality ) // sets alpha speed
7811{
7812 resize->fast_alpha = non_pma_alpha_speed_over_quality;
7813 resize->needs_rebuild = 1;
7814 return 1;
7815}
7816
7817STBIRDEF int stbir_set_input_subrect( STBIR_RESIZE * resize, double s0, double t0, double s1, double t1 ) // sets input region (full region by default)
7818{
7819 resize->input_s0 = s0;
7820 resize->input_t0 = t0;
7821 resize->input_s1 = s1;
7822 resize->input_t1 = t1;
7823 resize->needs_rebuild = 1;
7824
7825 // are we inbounds?
7826 if ( ( s1 < stbir__small_float ) || ( (s1-s0) < stbir__small_float ) ||
7827 ( t1 < stbir__small_float ) || ( (t1-t0) < stbir__small_float ) ||
7828 ( s0 > (1.0f-stbir__small_float) ) ||
7829 ( t0 > (1.0f-stbir__small_float) ) )
7830 return 0;
7831
7832 return 1;
7833}
7834
7835STBIRDEF int stbir_set_output_pixel_subrect( STBIR_RESIZE * resize, int subx, int suby, int subw, int subh ) // sets input region (full region by default)
7836{
7837 resize->output_subx = subx;
7838 resize->output_suby = suby;
7839 resize->output_subw = subw;
7840 resize->output_subh = subh;
7841 resize->needs_rebuild = 1;
7842
7843 // are we inbounds?
7844 if ( ( subx >= resize->output_w ) || ( ( subx + subw ) <= 0 ) || ( suby >= resize->output_h ) || ( ( suby + subh ) <= 0 ) || ( subw == 0 ) || ( subh == 0 ) )
7845 return 0;
7846
7847 return 1;
7848}
7849
7850STBIRDEF int stbir_set_pixel_subrect( STBIR_RESIZE * resize, int subx, int suby, int subw, int subh ) // sets both regions (full regions by default)
7851{
7852 double s0, t0, s1, t1;
7853
7854 s0 = ( (double)subx ) / ( (double)resize->output_w );
7855 t0 = ( (double)suby ) / ( (double)resize->output_h );
7856 s1 = ( (double)(subx+subw) ) / ( (double)resize->output_w );
7857 t1 = ( (double)(suby+subh) ) / ( (double)resize->output_h );
7858
7859 resize->input_s0 = s0;
7860 resize->input_t0 = t0;
7861 resize->input_s1 = s1;
7862 resize->input_t1 = t1;
7863 resize->output_subx = subx;
7864 resize->output_suby = suby;
7865 resize->output_subw = subw;
7866 resize->output_subh = subh;
7867 resize->needs_rebuild = 1;
7868
7869 // are we inbounds?
7870 if ( ( subx >= resize->output_w ) || ( ( subx + subw ) <= 0 ) || ( suby >= resize->output_h ) || ( ( suby + subh ) <= 0 ) || ( subw == 0 ) || ( subh == 0 ) )
7871 return 0;
7872
7873 return 1;
7874}
7875
7876static int stbir__perform_build( STBIR_RESIZE * resize, int splits )
7877{
7878 stbir__contributors conservative = { 0, 0 };
7879 stbir__sampler horizontal, vertical;
7880 int new_output_subx, new_output_suby;
7881 stbir__info * out_info;
7882 #ifdef STBIR_PROFILE
7883 stbir__info profile_infod; // used to contain building profile info before everything is allocated
7884 stbir__info * profile_info = &profile_infod;
7885 #endif
7886
7887 // have we already built the samplers?
7888 if ( resize->samplers )
7889 return 0;
7890
7891 #define STBIR_RETURN_ERROR_AND_ASSERT( exp ) STBIR_ASSERT( !(exp) ); if (exp) return 0;
7892 STBIR_RETURN_ERROR_AND_ASSERT( (unsigned)resize->horizontal_filter >= STBIR_FILTER_OTHER)
7893 STBIR_RETURN_ERROR_AND_ASSERT( (unsigned)resize->vertical_filter >= STBIR_FILTER_OTHER)
7894 #undef STBIR_RETURN_ERROR_AND_ASSERT
7895
7896 if ( splits <= 0 )
7897 return 0;
7898
7899 STBIR_PROFILE_BUILD_FIRST_START( build );
7900
7901 new_output_subx = resize->output_subx;
7902 new_output_suby = resize->output_suby;
7903
7904 // do horizontal clip and scale calcs
7905 if ( !stbir__calculate_region_transform( &horizontal.scale_info, resize->output_w, &new_output_subx, resize->output_subw, resize->input_w, resize->input_s0, resize->input_s1 ) )
7906 return 0;
7907
7908 // do vertical clip and scale calcs
7909 if ( !stbir__calculate_region_transform( &vertical.scale_info, resize->output_h, &new_output_suby, resize->output_subh, resize->input_h, resize->input_t0, resize->input_t1 ) )
7910 return 0;
7911
7912 // if nothing to do, just return
7913 if ( ( horizontal.scale_info.output_sub_size == 0 ) || ( vertical.scale_info.output_sub_size == 0 ) )
7914 return 0;
7915
7916 stbir__set_sampler(&horizontal, resize->horizontal_filter, resize->horizontal_filter_kernel, resize->horizontal_filter_support, resize->horizontal_edge, &horizontal.scale_info, 1, resize->user_data );
7917 stbir__get_conservative_extents( &horizontal, &conservative, resize->user_data );
7918 stbir__set_sampler(&vertical, resize->vertical_filter, resize->vertical_filter_kernel, resize->vertical_filter_support, resize->vertical_edge, &vertical.scale_info, 0, resize->user_data );
7919
7920 if ( ( vertical.scale_info.output_sub_size / splits ) < STBIR_FORCE_MINIMUM_SCANLINES_FOR_SPLITS ) // each split should be a minimum of 4 scanlines (handwavey choice)
7921 {
7922 splits = vertical.scale_info.output_sub_size / STBIR_FORCE_MINIMUM_SCANLINES_FOR_SPLITS;
7923 if ( splits == 0 ) splits = 1;
7924 }
7925
7926 STBIR_PROFILE_BUILD_START( alloc );
7927 out_info = stbir__alloc_internal_mem_and_build_samplers( &horizontal, &vertical, &conservative, resize->input_pixel_layout_public, resize->output_pixel_layout_public, splits, new_output_subx, new_output_suby, resize->fast_alpha, resize->user_data STBIR_ONLY_PROFILE_BUILD_SET_INFO );
7928 STBIR_PROFILE_BUILD_END( alloc );
7929 STBIR_PROFILE_BUILD_END( build );
7930
7931 if ( out_info )
7932 {
7933 resize->splits = splits;
7934 resize->samplers = out_info;
7935 resize->needs_rebuild = 0;
7936 #ifdef STBIR_PROFILE
7937 STBIR_MEMCPY( &out_info->profile, &profile_infod.profile, sizeof( out_info->profile ) );
7938 #endif
7939
7940 // update anything that can be changed without recalcing samplers
7941 stbir__update_info_from_resize( out_info, resize );
7942
7943 return splits;
7944 }
7945
7946 return 0;
7947}
7948
7949STBIRDEF void stbir_free_samplers( STBIR_RESIZE * resize )
7950{
7951 if ( resize->samplers )
7952 {
7953 stbir__free_internal_mem( resize->samplers );
7954 resize->samplers = 0;
7955 resize->called_alloc = 0;
7956 }
7957}
7958
7959STBIRDEF int stbir_build_samplers_with_splits( STBIR_RESIZE * resize, int splits )
7960{
7961 if ( ( resize->samplers == 0 ) || ( resize->needs_rebuild ) )
7962 {
7963 if ( resize->samplers )
7964 stbir_free_samplers( resize );
7965
7966 resize->called_alloc = 1;
7967 return stbir__perform_build( resize, splits );
7968 }
7969
7970 STBIR_PROFILE_BUILD_CLEAR( resize->samplers );
7971
7972 return 1;
7973}
7974
7975STBIRDEF int stbir_build_samplers( STBIR_RESIZE * resize )
7976{
7977 return stbir_build_samplers_with_splits( resize, 1 );
7978}
7979
7980STBIRDEF int stbir_resize_extended( STBIR_RESIZE * resize )
7981{
7982 int result;
7983
7984 if ( ( resize->samplers == 0 ) || ( resize->needs_rebuild ) )
7985 {
7986 int alloc_state = resize->called_alloc; // remember allocated state
7987
7988 if ( resize->samplers )
7989 {
7990 stbir__free_internal_mem( resize->samplers );
7991 resize->samplers = 0;
7992 }
7993
7994 if ( !stbir_build_samplers( resize ) )
7995 return 0;
7996
7997 resize->called_alloc = alloc_state;
7998
7999 // if build_samplers succeeded (above), but there are no samplers set, then
8000 // the area to stretch into was zero pixels, so don't do anything and return
8001 // success
8002 if ( resize->samplers == 0 )
8003 return 1;
8004 }
8005 else
8006 {
8007 // didn't build anything - clear it
8008 STBIR_PROFILE_BUILD_CLEAR( resize->samplers );
8009 }
8010
8011 // do resize
8012 result = stbir__perform_resize( resize->samplers, 0, resize->splits );
8013
8014 // if we alloced, then free
8015 if ( !resize->called_alloc )
8016 {
8017 stbir_free_samplers( resize );
8018 resize->samplers = 0;
8019 }
8020
8021 return result;
8022}
8023
8024STBIRDEF int stbir_resize_extended_split( STBIR_RESIZE * resize, int split_start, int split_count )
8025{
8026 STBIR_ASSERT( resize->samplers );
8027
8028 // if we're just doing the whole thing, call full
8029 if ( ( split_start == -1 ) || ( ( split_start == 0 ) && ( split_count == resize->splits ) ) )
8030 return stbir_resize_extended( resize );
8031
8032 // you **must** build samplers first when using split resize
8033 if ( ( resize->samplers == 0 ) || ( resize->needs_rebuild ) )
8034 return 0;
8035
8036 if ( ( split_start >= resize->splits ) || ( split_start < 0 ) || ( ( split_start + split_count ) > resize->splits ) || ( split_count <= 0 ) )
8037 return 0;
8038
8039 // do resize
8040 return stbir__perform_resize( resize->samplers, split_start, split_count );
8041}
8042
8043
8044static void * stbir_quick_resize_helper( const void *input_pixels , int input_w , int input_h, int input_stride_in_bytes,
8045 void *output_pixels, int output_w, int output_h, int output_stride_in_bytes,
8046 stbir_pixel_layout pixel_layout, stbir_datatype data_type, stbir_edge edge, stbir_filter filter )
8047{
8048 STBIR_RESIZE resize;
8049 int scanline_output_in_bytes;
8050 int positive_output_stride_in_bytes;
8051 void * start_ptr;
8052 void * free_ptr;
8053
8054 scanline_output_in_bytes = output_w * stbir__type_size[ data_type ] * stbir__pixel_channels[ stbir__pixel_layout_convert_public_to_internal[ pixel_layout ] ];
8055 if ( scanline_output_in_bytes == 0 )
8056 return 0;
8057
8058 // if zero stride, use scanline output
8059 if ( output_stride_in_bytes == 0 )
8060 output_stride_in_bytes = scanline_output_in_bytes;
8061
8062 // abs value for inverted images (negative pitches)
8063 positive_output_stride_in_bytes = output_stride_in_bytes;
8064 if ( positive_output_stride_in_bytes < 0 )
8065 positive_output_stride_in_bytes = -positive_output_stride_in_bytes;
8066
8067 // is the requested stride smaller than the scanline output? if so, just fail
8068 if ( positive_output_stride_in_bytes < scanline_output_in_bytes )
8069 return 0;
8070
8071 start_ptr = output_pixels;
8072 free_ptr = 0; // no free pointer, since they passed buffer to use
8073
8074 // did they pass a zero for the dest? if so, allocate the buffer
8075 if ( output_pixels == 0 )
8076 {
8077 size_t size;
8078 char * ptr;
8079
8080 size = (size_t)positive_output_stride_in_bytes * (size_t)output_h;
8081 if ( size == 0 )
8082 return 0;
8083
8084 ptr = (char*) STBIR_MALLOC( size, 0 );
8085 if ( ptr == 0 )
8086 return 0;
8087
8088 free_ptr = ptr;
8089
8090 // point at the last scanline, if they requested a flipped image
8091 if ( output_stride_in_bytes < 0 )
8092 start_ptr = ptr + ( (size_t)positive_output_stride_in_bytes * (size_t)( output_h - 1 ) );
8093 else
8094 start_ptr = ptr;
8095 }
8096
8097 // ok, now do the resize
8098 stbir_resize_init( &resize,
8099 input_pixels, input_w, input_h, input_stride_in_bytes,
8100 start_ptr, output_w, output_h, output_stride_in_bytes,
8101 pixel_layout, data_type );
8102
8103 resize.horizontal_edge = edge;
8104 resize.vertical_edge = edge;
8105 resize.horizontal_filter = filter;
8106 resize.vertical_filter = filter;
8107
8108 if ( !stbir_resize_extended( &resize ) )
8109 {
8110 if ( free_ptr )
8111 STBIR_FREE( free_ptr, 0 );
8112 return 0;
8113 }
8114
8115 return (free_ptr) ? free_ptr : start_ptr;
8116}
8117
8118
8119
8120STBIRDEF unsigned char * stbir_resize_uint8_linear( const unsigned char *input_pixels , int input_w , int input_h, int input_stride_in_bytes,
8121 unsigned char *output_pixels, int output_w, int output_h, int output_stride_in_bytes,
8122 stbir_pixel_layout pixel_layout )
8123{
8124 return (unsigned char *) stbir_quick_resize_helper( input_pixels , input_w , input_h, input_stride_in_bytes,
8125 output_pixels, output_w, output_h, output_stride_in_bytes,
8126 pixel_layout, STBIR_TYPE_UINT8, STBIR_EDGE_CLAMP, STBIR_FILTER_DEFAULT );
8127}
8128
8129STBIRDEF unsigned char * stbir_resize_uint8_srgb( const unsigned char *input_pixels , int input_w , int input_h, int input_stride_in_bytes,
8130 unsigned char *output_pixels, int output_w, int output_h, int output_stride_in_bytes,
8131 stbir_pixel_layout pixel_layout )
8132{
8133 return (unsigned char *) stbir_quick_resize_helper( input_pixels , input_w , input_h, input_stride_in_bytes,
8134 output_pixels, output_w, output_h, output_stride_in_bytes,
8135 pixel_layout, STBIR_TYPE_UINT8_SRGB, STBIR_EDGE_CLAMP, STBIR_FILTER_DEFAULT );
8136}
8137
8138
8139STBIRDEF float * stbir_resize_float_linear( const float *input_pixels , int input_w , int input_h, int input_stride_in_bytes,
8140 float *output_pixels, int output_w, int output_h, int output_stride_in_bytes,
8141 stbir_pixel_layout pixel_layout )
8142{
8143 return (float *) stbir_quick_resize_helper( input_pixels , input_w , input_h, input_stride_in_bytes,
8144 output_pixels, output_w, output_h, output_stride_in_bytes,
8145 pixel_layout, STBIR_TYPE_FLOAT, STBIR_EDGE_CLAMP, STBIR_FILTER_DEFAULT );
8146}
8147
8148
8149STBIRDEF void * stbir_resize( const void *input_pixels , int input_w , int input_h, int input_stride_in_bytes,
8150 void *output_pixels, int output_w, int output_h, int output_stride_in_bytes,
8151 stbir_pixel_layout pixel_layout, stbir_datatype data_type,
8152 stbir_edge edge, stbir_filter filter )
8153{
8154 return (void *) stbir_quick_resize_helper( input_pixels , input_w , input_h, input_stride_in_bytes,
8155 output_pixels, output_w, output_h, output_stride_in_bytes,
8156 pixel_layout, data_type, edge, filter );
8157}
8158
8159#ifdef STBIR_PROFILE
8160
8161STBIRDEF void stbir_resize_build_profile_info( STBIR_PROFILE_INFO * info, STBIR_RESIZE const * resize )
8162{
8163 static char const * bdescriptions[6] = { "Building", "Allocating", "Horizontal sampler", "Vertical sampler", "Coefficient cleanup", "Coefficient pivot" } ;
8164 stbir__info* samp = resize->samplers;
8165 int i;
8166
8167 typedef int testa[ (STBIR__ARRAY_SIZE( bdescriptions ) == (STBIR__ARRAY_SIZE( samp->profile.array )-1) )?1:-1];
8168 typedef int testb[ (sizeof( samp->profile.array ) == (sizeof(samp->profile.named)) )?1:-1];
8169 typedef int testc[ (sizeof( info->clocks ) >= (sizeof(samp->profile.named)) )?1:-1];
8170
8171 for( i = 0 ; i < STBIR__ARRAY_SIZE( bdescriptions ) ; i++)
8172 info->clocks[i] = samp->profile.array[i+1];
8173
8174 info->total_clocks = samp->profile.named.total;
8175 info->descriptions = bdescriptions;
8176 info->count = STBIR__ARRAY_SIZE( bdescriptions );
8177}
8178
8179STBIRDEF void stbir_resize_split_profile_info( STBIR_PROFILE_INFO * info, STBIR_RESIZE const * resize, int split_start, int split_count )
8180{
8181 static char const * descriptions[7] = { "Looping", "Vertical sampling", "Horizontal sampling", "Scanline input", "Scanline output", "Alpha weighting", "Alpha unweighting" };
8182 stbir__per_split_info * split_info;
8183 int s, i;
8184
8185 typedef int testa[ (STBIR__ARRAY_SIZE( descriptions ) == (STBIR__ARRAY_SIZE( split_info->profile.array )-1) )?1:-1];
8186 typedef int testb[ (sizeof( split_info->profile.array ) == (sizeof(split_info->profile.named)) )?1:-1];
8187 typedef int testc[ (sizeof( info->clocks ) >= (sizeof(split_info->profile.named)) )?1:-1];
8188
8189 if ( split_start == -1 )
8190 {
8191 split_start = 0;
8192 split_count = resize->samplers->splits;
8193 }
8194
8195 if ( ( split_start >= resize->splits ) || ( split_start < 0 ) || ( ( split_start + split_count ) > resize->splits ) || ( split_count <= 0 ) )
8196 {
8197 info->total_clocks = 0;
8198 info->descriptions = 0;
8199 info->count = 0;
8200 return;
8201 }
8202
8203 split_info = resize->samplers->split_info + split_start;
8204
8205 // sum up the profile from all the splits
8206 for( i = 0 ; i < STBIR__ARRAY_SIZE( descriptions ) ; i++ )
8207 {
8208 stbir_uint64 sum = 0;
8209 for( s = 0 ; s < split_count ; s++ )
8210 sum += split_info[s].profile.array[i+1];
8211 info->clocks[i] = sum;
8212 }
8213
8214 info->total_clocks = split_info->profile.named.total;
8215 info->descriptions = descriptions;
8216 info->count = STBIR__ARRAY_SIZE( descriptions );
8217}
8218
8219STBIRDEF void stbir_resize_extended_profile_info( STBIR_PROFILE_INFO * info, STBIR_RESIZE const * resize )
8220{
8221 stbir_resize_split_profile_info( info, resize, -1, 0 );
8222}
8223
8224#endif // STBIR_PROFILE
8225
8226#undef STBIR_BGR
8227#undef STBIR_1CHANNEL
8228#undef STBIR_2CHANNEL
8229#undef STBIR_RGB
8230#undef STBIR_RGBA
8231#undef STBIR_4CHANNEL
8232#undef STBIR_BGRA
8233#undef STBIR_ARGB
8234#undef STBIR_ABGR
8235#undef STBIR_RA
8236#undef STBIR_AR
8237#undef STBIR_RGBA_PM
8238#undef STBIR_BGRA_PM
8239#undef STBIR_ARGB_PM
8240#undef STBIR_ABGR_PM
8241#undef STBIR_RA_PM
8242#undef STBIR_AR_PM
8243
8244#endif // STB_IMAGE_RESIZE_IMPLEMENTATION
8245
8246#else // STB_IMAGE_RESIZE_HORIZONTALS&STB_IMAGE_RESIZE_DO_VERTICALS
8247
8248// we reinclude the header file to define all the horizontal functions
8249// specializing each function for the number of coeffs is 20-40% faster *OVERALL*
8250
8251// by including the header file again this way, we can still debug the functions
8252
8253#define STBIR_strs_join2( start, mid, end ) start##mid##end
8254#define STBIR_strs_join1( start, mid, end ) STBIR_strs_join2( start, mid, end )
8255
8256#define STBIR_strs_join24( start, mid1, mid2, end ) start##mid1##mid2##end
8257#define STBIR_strs_join14( start, mid1, mid2, end ) STBIR_strs_join24( start, mid1, mid2, end )
8258
8259#ifdef STB_IMAGE_RESIZE_DO_CODERS
8260
8261#ifdef stbir__decode_suffix
8262#define STBIR__CODER_NAME( name ) STBIR_strs_join1( name, _, stbir__decode_suffix )
8263#else
8264#define STBIR__CODER_NAME( name ) name
8265#endif
8266
8267#ifdef stbir__decode_swizzle
8268#define stbir__decode_simdf8_flip(reg) STBIR_strs_join1( STBIR_strs_join1( STBIR_strs_join1( STBIR_strs_join1( stbir__simdf8_0123to,stbir__decode_order0,stbir__decode_order1),stbir__decode_order2,stbir__decode_order3),stbir__decode_order0,stbir__decode_order1),stbir__decode_order2,stbir__decode_order3)(reg, reg)
8269#define stbir__decode_simdf4_flip(reg) STBIR_strs_join1( STBIR_strs_join1( stbir__simdf_0123to,stbir__decode_order0,stbir__decode_order1),stbir__decode_order2,stbir__decode_order3)(reg, reg)
8270#define stbir__encode_simdf8_unflip(reg) STBIR_strs_join1( STBIR_strs_join1( STBIR_strs_join1( STBIR_strs_join1( stbir__simdf8_0123to,stbir__encode_order0,stbir__encode_order1),stbir__encode_order2,stbir__encode_order3),stbir__encode_order0,stbir__encode_order1),stbir__encode_order2,stbir__encode_order3)(reg, reg)
8271#define stbir__encode_simdf4_unflip(reg) STBIR_strs_join1( STBIR_strs_join1( stbir__simdf_0123to,stbir__encode_order0,stbir__encode_order1),stbir__encode_order2,stbir__encode_order3)(reg, reg)
8272#else
8273#define stbir__decode_order0 0
8274#define stbir__decode_order1 1
8275#define stbir__decode_order2 2
8276#define stbir__decode_order3 3
8277#define stbir__encode_order0 0
8278#define stbir__encode_order1 1
8279#define stbir__encode_order2 2
8280#define stbir__encode_order3 3
8281#define stbir__decode_simdf8_flip(reg)
8282#define stbir__decode_simdf4_flip(reg)
8283#define stbir__encode_simdf8_unflip(reg)
8284#define stbir__encode_simdf4_unflip(reg)
8285#endif
8286
8287#ifdef STBIR_SIMD8
8288#define stbir__encode_simdfX_unflip stbir__encode_simdf8_unflip
8289#else
8290#define stbir__encode_simdfX_unflip stbir__encode_simdf4_unflip
8291#endif
8292
8293static float * STBIR__CODER_NAME( stbir__decode_uint8_linear_scaled )( float * decodep, int width_times_channels, void const * inputp )
8294{
8295 float STBIR_STREAMOUT_PTR( * ) decode = decodep;
8296 float * decode_end = (float*) decode + width_times_channels;
8297 unsigned char const * input = (unsigned char const*)inputp;
8298
8299 #ifdef STBIR_SIMD
8300 unsigned char const * end_input_m16 = input + width_times_channels - 16;
8301 if ( width_times_channels >= 16 )
8302 {
8303 decode_end -= 16;
8304 STBIR_NO_UNROLL_LOOP_START_INF_FOR
8305 for(;;)
8306 {
8307 #ifdef STBIR_SIMD8
8308 stbir__simdi i; stbir__simdi8 o0,o1;
8309 stbir__simdf8 of0, of1;
8310 STBIR_NO_UNROLL(decode);
8311 stbir__simdi_load( i, input );
8312 stbir__simdi8_expand_u8_to_u32( o0, o1, i );
8313 stbir__simdi8_convert_i32_to_float( of0, o0 );
8314 stbir__simdi8_convert_i32_to_float( of1, o1 );
8315 stbir__simdf8_mult( of0, of0, STBIR_max_uint8_as_float_inverted8);
8316 stbir__simdf8_mult( of1, of1, STBIR_max_uint8_as_float_inverted8);
8317 stbir__decode_simdf8_flip( of0 );
8318 stbir__decode_simdf8_flip( of1 );
8319 stbir__simdf8_store( decode + 0, of0 );
8320 stbir__simdf8_store( decode + 8, of1 );
8321 #else
8322 stbir__simdi i, o0, o1, o2, o3;
8323 stbir__simdf of0, of1, of2, of3;
8324 STBIR_NO_UNROLL(decode);
8325 stbir__simdi_load( i, input );
8326 stbir__simdi_expand_u8_to_u32( o0,o1,o2,o3,i);
8327 stbir__simdi_convert_i32_to_float( of0, o0 );
8328 stbir__simdi_convert_i32_to_float( of1, o1 );
8329 stbir__simdi_convert_i32_to_float( of2, o2 );
8330 stbir__simdi_convert_i32_to_float( of3, o3 );
8331 stbir__simdf_mult( of0, of0, STBIR__CONSTF(STBIR_max_uint8_as_float_inverted) );
8332 stbir__simdf_mult( of1, of1, STBIR__CONSTF(STBIR_max_uint8_as_float_inverted) );
8333 stbir__simdf_mult( of2, of2, STBIR__CONSTF(STBIR_max_uint8_as_float_inverted) );
8334 stbir__simdf_mult( of3, of3, STBIR__CONSTF(STBIR_max_uint8_as_float_inverted) );
8335 stbir__decode_simdf4_flip( of0 );
8336 stbir__decode_simdf4_flip( of1 );
8337 stbir__decode_simdf4_flip( of2 );
8338 stbir__decode_simdf4_flip( of3 );
8339 stbir__simdf_store( decode + 0, of0 );
8340 stbir__simdf_store( decode + 4, of1 );
8341 stbir__simdf_store( decode + 8, of2 );
8342 stbir__simdf_store( decode + 12, of3 );
8343 #endif
8344 decode += 16;
8345 input += 16;
8346 if ( decode <= decode_end )
8347 continue;
8348 if ( decode == ( decode_end + 16 ) )
8349 break;
8350 decode = decode_end; // backup and do last couple
8351 input = end_input_m16;
8352 }
8353 return decode_end + 16;
8354 }
8355 #endif
8356
8357 // try to do blocks of 4 when you can
8358 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
8359 decode += 4;
8360 STBIR_SIMD_NO_UNROLL_LOOP_START
8361 while( decode <= decode_end )
8362 {
8363 STBIR_SIMD_NO_UNROLL(decode);
8364 decode[0-4] = ((float)(input[stbir__decode_order0])) * stbir__max_uint8_as_float_inverted;
8365 decode[1-4] = ((float)(input[stbir__decode_order1])) * stbir__max_uint8_as_float_inverted;
8366 decode[2-4] = ((float)(input[stbir__decode_order2])) * stbir__max_uint8_as_float_inverted;
8367 decode[3-4] = ((float)(input[stbir__decode_order3])) * stbir__max_uint8_as_float_inverted;
8368 decode += 4;
8369 input += 4;
8370 }
8371 decode -= 4;
8372 #endif
8373
8374 // do the remnants
8375 #if stbir__coder_min_num < 4
8376 STBIR_NO_UNROLL_LOOP_START
8377 while( decode < decode_end )
8378 {
8379 STBIR_NO_UNROLL(decode);
8380 decode[0] = ((float)(input[stbir__decode_order0])) * stbir__max_uint8_as_float_inverted;
8381 #if stbir__coder_min_num >= 2
8382 decode[1] = ((float)(input[stbir__decode_order1])) * stbir__max_uint8_as_float_inverted;
8383 #endif
8384 #if stbir__coder_min_num >= 3
8385 decode[2] = ((float)(input[stbir__decode_order2])) * stbir__max_uint8_as_float_inverted;
8386 #endif
8387 decode += stbir__coder_min_num;
8388 input += stbir__coder_min_num;
8389 }
8390 #endif
8391
8392 return decode_end;
8393}
8394
8395static void STBIR__CODER_NAME( stbir__encode_uint8_linear_scaled )( void * outputp, int width_times_channels, float const * encode )
8396{
8397 unsigned char STBIR_SIMD_STREAMOUT_PTR( * ) output = (unsigned char *) outputp;
8398 unsigned char * end_output = ( (unsigned char *) output ) + width_times_channels;
8399
8400 #ifdef STBIR_SIMD
8401 if ( width_times_channels >= stbir__simdfX_float_count*2 )
8402 {
8403 float const * end_encode_m8 = encode + width_times_channels - stbir__simdfX_float_count*2;
8404 end_output -= stbir__simdfX_float_count*2;
8405 STBIR_NO_UNROLL_LOOP_START_INF_FOR
8406 for(;;)
8407 {
8408 stbir__simdfX e0, e1;
8409 stbir__simdi i;
8410 STBIR_SIMD_NO_UNROLL(encode);
8411 stbir__simdfX_madd_mem( e0, STBIR_simd_point5X, STBIR_max_uint8_as_floatX, encode );
8412 stbir__simdfX_madd_mem( e1, STBIR_simd_point5X, STBIR_max_uint8_as_floatX, encode+stbir__simdfX_float_count );
8413 stbir__encode_simdfX_unflip( e0 );
8414 stbir__encode_simdfX_unflip( e1 );
8415 #ifdef STBIR_SIMD8
8416 stbir__simdf8_pack_to_16bytes( i, e0, e1 );
8417 stbir__simdi_store( output, i );
8418 #else
8419 stbir__simdf_pack_to_8bytes( i, e0, e1 );
8420 stbir__simdi_store2( output, i );
8421 #endif
8422 encode += stbir__simdfX_float_count*2;
8423 output += stbir__simdfX_float_count*2;
8424 if ( output <= end_output )
8425 continue;
8426 if ( output == ( end_output + stbir__simdfX_float_count*2 ) )
8427 break;
8428 output = end_output; // backup and do last couple
8429 encode = end_encode_m8;
8430 }
8431 return;
8432 }
8433
8434 // try to do blocks of 4 when you can
8435 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
8436 output += 4;
8437 STBIR_NO_UNROLL_LOOP_START
8438 while( output <= end_output )
8439 {
8440 stbir__simdf e0;
8441 stbir__simdi i0;
8442 STBIR_NO_UNROLL(encode);
8443 stbir__simdf_load( e0, encode );
8444 stbir__simdf_madd( e0, STBIR__CONSTF(STBIR_simd_point5), STBIR__CONSTF(STBIR_max_uint8_as_float), e0 );
8445 stbir__encode_simdf4_unflip( e0 );
8446 stbir__simdf_pack_to_8bytes( i0, e0, e0 ); // only use first 4
8447 *(int*)(output-4) = stbir__simdi_to_int( i0 );
8448 output += 4;
8449 encode += 4;
8450 }
8451 output -= 4;
8452 #endif
8453
8454 // do the remnants
8455 #if stbir__coder_min_num < 4
8456 STBIR_NO_UNROLL_LOOP_START
8457 while( output < end_output )
8458 {
8459 stbir__simdf e0;
8460 STBIR_NO_UNROLL(encode);
8461 stbir__simdf_madd1_mem( e0, STBIR__CONSTF(STBIR_simd_point5), STBIR__CONSTF(STBIR_max_uint8_as_float), encode+stbir__encode_order0 ); output[0] = stbir__simdf_convert_float_to_uint8( e0 );
8462 #if stbir__coder_min_num >= 2
8463 stbir__simdf_madd1_mem( e0, STBIR__CONSTF(STBIR_simd_point5), STBIR__CONSTF(STBIR_max_uint8_as_float), encode+stbir__encode_order1 ); output[1] = stbir__simdf_convert_float_to_uint8( e0 );
8464 #endif
8465 #if stbir__coder_min_num >= 3
8466 stbir__simdf_madd1_mem( e0, STBIR__CONSTF(STBIR_simd_point5), STBIR__CONSTF(STBIR_max_uint8_as_float), encode+stbir__encode_order2 ); output[2] = stbir__simdf_convert_float_to_uint8( e0 );
8467 #endif
8468 output += stbir__coder_min_num;
8469 encode += stbir__coder_min_num;
8470 }
8471 #endif
8472
8473 #else
8474
8475 // try to do blocks of 4 when you can
8476 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
8477 output += 4;
8478 while( output <= end_output )
8479 {
8480 float f;
8481 f = encode[stbir__encode_order0] * stbir__max_uint8_as_float + 0.5f; STBIR_CLAMP(f, 0, 255); output[0-4] = (unsigned char)f;
8482 f = encode[stbir__encode_order1] * stbir__max_uint8_as_float + 0.5f; STBIR_CLAMP(f, 0, 255); output[1-4] = (unsigned char)f;
8483 f = encode[stbir__encode_order2] * stbir__max_uint8_as_float + 0.5f; STBIR_CLAMP(f, 0, 255); output[2-4] = (unsigned char)f;
8484 f = encode[stbir__encode_order3] * stbir__max_uint8_as_float + 0.5f; STBIR_CLAMP(f, 0, 255); output[3-4] = (unsigned char)f;
8485 output += 4;
8486 encode += 4;
8487 }
8488 output -= 4;
8489 #endif
8490
8491 // do the remnants
8492 #if stbir__coder_min_num < 4
8493 STBIR_NO_UNROLL_LOOP_START
8494 while( output < end_output )
8495 {
8496 float f;
8497 STBIR_NO_UNROLL(encode);
8498 f = encode[stbir__encode_order0] * stbir__max_uint8_as_float + 0.5f; STBIR_CLAMP(f, 0, 255); output[0] = (unsigned char)f;
8499 #if stbir__coder_min_num >= 2
8500 f = encode[stbir__encode_order1] * stbir__max_uint8_as_float + 0.5f; STBIR_CLAMP(f, 0, 255); output[1] = (unsigned char)f;
8501 #endif
8502 #if stbir__coder_min_num >= 3
8503 f = encode[stbir__encode_order2] * stbir__max_uint8_as_float + 0.5f; STBIR_CLAMP(f, 0, 255); output[2] = (unsigned char)f;
8504 #endif
8505 output += stbir__coder_min_num;
8506 encode += stbir__coder_min_num;
8507 }
8508 #endif
8509 #endif
8510}
8511
8512static float * STBIR__CODER_NAME(stbir__decode_uint8_linear)( float * decodep, int width_times_channels, void const * inputp )
8513{
8514 float STBIR_STREAMOUT_PTR( * ) decode = decodep;
8515 float * decode_end = (float*) decode + width_times_channels;
8516 unsigned char const * input = (unsigned char const*)inputp;
8517
8518 #ifdef STBIR_SIMD
8519 unsigned char const * end_input_m16 = input + width_times_channels - 16;
8520 if ( width_times_channels >= 16 )
8521 {
8522 decode_end -= 16;
8523 STBIR_NO_UNROLL_LOOP_START_INF_FOR
8524 for(;;)
8525 {
8526 #ifdef STBIR_SIMD8
8527 stbir__simdi i; stbir__simdi8 o0,o1;
8528 stbir__simdf8 of0, of1;
8529 STBIR_NO_UNROLL(decode);
8530 stbir__simdi_load( i, input );
8531 stbir__simdi8_expand_u8_to_u32( o0, o1, i );
8532 stbir__simdi8_convert_i32_to_float( of0, o0 );
8533 stbir__simdi8_convert_i32_to_float( of1, o1 );
8534 stbir__decode_simdf8_flip( of0 );
8535 stbir__decode_simdf8_flip( of1 );
8536 stbir__simdf8_store( decode + 0, of0 );
8537 stbir__simdf8_store( decode + 8, of1 );
8538 #else
8539 stbir__simdi i, o0, o1, o2, o3;
8540 stbir__simdf of0, of1, of2, of3;
8541 STBIR_NO_UNROLL(decode);
8542 stbir__simdi_load( i, input );
8543 stbir__simdi_expand_u8_to_u32( o0,o1,o2,o3,i);
8544 stbir__simdi_convert_i32_to_float( of0, o0 );
8545 stbir__simdi_convert_i32_to_float( of1, o1 );
8546 stbir__simdi_convert_i32_to_float( of2, o2 );
8547 stbir__simdi_convert_i32_to_float( of3, o3 );
8548 stbir__decode_simdf4_flip( of0 );
8549 stbir__decode_simdf4_flip( of1 );
8550 stbir__decode_simdf4_flip( of2 );
8551 stbir__decode_simdf4_flip( of3 );
8552 stbir__simdf_store( decode + 0, of0 );
8553 stbir__simdf_store( decode + 4, of1 );
8554 stbir__simdf_store( decode + 8, of2 );
8555 stbir__simdf_store( decode + 12, of3 );
8556#endif
8557 decode += 16;
8558 input += 16;
8559 if ( decode <= decode_end )
8560 continue;
8561 if ( decode == ( decode_end + 16 ) )
8562 break;
8563 decode = decode_end; // backup and do last couple
8564 input = end_input_m16;
8565 }
8566 return decode_end + 16;
8567 }
8568 #endif
8569
8570 // try to do blocks of 4 when you can
8571 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
8572 decode += 4;
8573 STBIR_SIMD_NO_UNROLL_LOOP_START
8574 while( decode <= decode_end )
8575 {
8576 STBIR_SIMD_NO_UNROLL(decode);
8577 decode[0-4] = ((float)(input[stbir__decode_order0]));
8578 decode[1-4] = ((float)(input[stbir__decode_order1]));
8579 decode[2-4] = ((float)(input[stbir__decode_order2]));
8580 decode[3-4] = ((float)(input[stbir__decode_order3]));
8581 decode += 4;
8582 input += 4;
8583 }
8584 decode -= 4;
8585 #endif
8586
8587 // do the remnants
8588 #if stbir__coder_min_num < 4
8589 STBIR_NO_UNROLL_LOOP_START
8590 while( decode < decode_end )
8591 {
8592 STBIR_NO_UNROLL(decode);
8593 decode[0] = ((float)(input[stbir__decode_order0]));
8594 #if stbir__coder_min_num >= 2
8595 decode[1] = ((float)(input[stbir__decode_order1]));
8596 #endif
8597 #if stbir__coder_min_num >= 3
8598 decode[2] = ((float)(input[stbir__decode_order2]));
8599 #endif
8600 decode += stbir__coder_min_num;
8601 input += stbir__coder_min_num;
8602 }
8603 #endif
8604 return decode_end;
8605}
8606
8607static void STBIR__CODER_NAME( stbir__encode_uint8_linear )( void * outputp, int width_times_channels, float const * encode )
8608{
8609 unsigned char STBIR_SIMD_STREAMOUT_PTR( * ) output = (unsigned char *) outputp;
8610 unsigned char * end_output = ( (unsigned char *) output ) + width_times_channels;
8611
8612 #ifdef STBIR_SIMD
8613 if ( width_times_channels >= stbir__simdfX_float_count*2 )
8614 {
8615 float const * end_encode_m8 = encode + width_times_channels - stbir__simdfX_float_count*2;
8616 end_output -= stbir__simdfX_float_count*2;
8617 STBIR_SIMD_NO_UNROLL_LOOP_START_INF_FOR
8618 for(;;)
8619 {
8620 stbir__simdfX e0, e1;
8621 stbir__simdi i;
8622 STBIR_SIMD_NO_UNROLL(encode);
8623 stbir__simdfX_add_mem( e0, STBIR_simd_point5X, encode );
8624 stbir__simdfX_add_mem( e1, STBIR_simd_point5X, encode+stbir__simdfX_float_count );
8625 stbir__encode_simdfX_unflip( e0 );
8626 stbir__encode_simdfX_unflip( e1 );
8627 #ifdef STBIR_SIMD8
8628 stbir__simdf8_pack_to_16bytes( i, e0, e1 );
8629 stbir__simdi_store( output, i );
8630 #else
8631 stbir__simdf_pack_to_8bytes( i, e0, e1 );
8632 stbir__simdi_store2( output, i );
8633 #endif
8634 encode += stbir__simdfX_float_count*2;
8635 output += stbir__simdfX_float_count*2;
8636 if ( output <= end_output )
8637 continue;
8638 if ( output == ( end_output + stbir__simdfX_float_count*2 ) )
8639 break;
8640 output = end_output; // backup and do last couple
8641 encode = end_encode_m8;
8642 }
8643 return;
8644 }
8645
8646 // try to do blocks of 4 when you can
8647 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
8648 output += 4;
8649 STBIR_NO_UNROLL_LOOP_START
8650 while( output <= end_output )
8651 {
8652 stbir__simdf e0;
8653 stbir__simdi i0;
8654 STBIR_NO_UNROLL(encode);
8655 stbir__simdf_load( e0, encode );
8656 stbir__simdf_add( e0, STBIR__CONSTF(STBIR_simd_point5), e0 );
8657 stbir__encode_simdf4_unflip( e0 );
8658 stbir__simdf_pack_to_8bytes( i0, e0, e0 ); // only use first 4
8659 *(int*)(output-4) = stbir__simdi_to_int( i0 );
8660 output += 4;
8661 encode += 4;
8662 }
8663 output -= 4;
8664 #endif
8665
8666 #else
8667
8668 // try to do blocks of 4 when you can
8669 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
8670 output += 4;
8671 while( output <= end_output )
8672 {
8673 float f;
8674 f = encode[stbir__encode_order0] + 0.5f; STBIR_CLAMP(f, 0, 255); output[0-4] = (unsigned char)f;
8675 f = encode[stbir__encode_order1] + 0.5f; STBIR_CLAMP(f, 0, 255); output[1-4] = (unsigned char)f;
8676 f = encode[stbir__encode_order2] + 0.5f; STBIR_CLAMP(f, 0, 255); output[2-4] = (unsigned char)f;
8677 f = encode[stbir__encode_order3] + 0.5f; STBIR_CLAMP(f, 0, 255); output[3-4] = (unsigned char)f;
8678 output += 4;
8679 encode += 4;
8680 }
8681 output -= 4;
8682 #endif
8683
8684 #endif
8685
8686 // do the remnants
8687 #if stbir__coder_min_num < 4
8688 STBIR_NO_UNROLL_LOOP_START
8689 while( output < end_output )
8690 {
8691 float f;
8692 STBIR_NO_UNROLL(encode);
8693 f = encode[stbir__encode_order0] + 0.5f; STBIR_CLAMP(f, 0, 255); output[0] = (unsigned char)f;
8694 #if stbir__coder_min_num >= 2
8695 f = encode[stbir__encode_order1] + 0.5f; STBIR_CLAMP(f, 0, 255); output[1] = (unsigned char)f;
8696 #endif
8697 #if stbir__coder_min_num >= 3
8698 f = encode[stbir__encode_order2] + 0.5f; STBIR_CLAMP(f, 0, 255); output[2] = (unsigned char)f;
8699 #endif
8700 output += stbir__coder_min_num;
8701 encode += stbir__coder_min_num;
8702 }
8703 #endif
8704}
8705
8706static float * STBIR__CODER_NAME(stbir__decode_uint8_srgb)( float * decodep, int width_times_channels, void const * inputp )
8707{
8708 float STBIR_STREAMOUT_PTR( * ) decode = decodep;
8709 float * decode_end = (float*) decode + width_times_channels;
8710 unsigned char const * input = (unsigned char const *)inputp;
8711
8712 // try to do blocks of 4 when you can
8713 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
8714 decode += 4;
8715 while( decode <= decode_end )
8716 {
8717 decode[0-4] = stbir__srgb_uchar_to_linear_float[ input[ stbir__decode_order0 ] ];
8718 decode[1-4] = stbir__srgb_uchar_to_linear_float[ input[ stbir__decode_order1 ] ];
8719 decode[2-4] = stbir__srgb_uchar_to_linear_float[ input[ stbir__decode_order2 ] ];
8720 decode[3-4] = stbir__srgb_uchar_to_linear_float[ input[ stbir__decode_order3 ] ];
8721 decode += 4;
8722 input += 4;
8723 }
8724 decode -= 4;
8725 #endif
8726
8727 // do the remnants
8728 #if stbir__coder_min_num < 4
8729 STBIR_NO_UNROLL_LOOP_START
8730 while( decode < decode_end )
8731 {
8732 STBIR_NO_UNROLL(decode);
8733 decode[0] = stbir__srgb_uchar_to_linear_float[ input[ stbir__decode_order0 ] ];
8734 #if stbir__coder_min_num >= 2
8735 decode[1] = stbir__srgb_uchar_to_linear_float[ input[ stbir__decode_order1 ] ];
8736 #endif
8737 #if stbir__coder_min_num >= 3
8738 decode[2] = stbir__srgb_uchar_to_linear_float[ input[ stbir__decode_order2 ] ];
8739 #endif
8740 decode += stbir__coder_min_num;
8741 input += stbir__coder_min_num;
8742 }
8743 #endif
8744 return decode_end;
8745}
8746
8747#define stbir__min_max_shift20( i, f ) \
8748 stbir__simdf_max( f, f, stbir_simdf_casti(STBIR__CONSTI( STBIR_almost_zero )) ); \
8749 stbir__simdf_min( f, f, stbir_simdf_casti(STBIR__CONSTI( STBIR_almost_one )) ); \
8750 stbir__simdi_32shr( i, stbir_simdi_castf( f ), 20 );
8751
8752#define stbir__scale_and_convert( i, f ) \
8753 stbir__simdf_madd( f, STBIR__CONSTF( STBIR_simd_point5 ), STBIR__CONSTF( STBIR_max_uint8_as_float ), f ); \
8754 stbir__simdf_max( f, f, stbir__simdf_zeroP() ); \
8755 stbir__simdf_min( f, f, STBIR__CONSTF( STBIR_max_uint8_as_float ) ); \
8756 stbir__simdf_convert_float_to_i32( i, f );
8757
8758#define stbir__linear_to_srgb_finish( i, f ) \
8759{ \
8760 stbir__simdi temp; \
8761 stbir__simdi_32shr( temp, stbir_simdi_castf( f ), 12 ) ; \
8762 stbir__simdi_and( temp, temp, STBIR__CONSTI(STBIR_mantissa_mask) ); \
8763 stbir__simdi_or( temp, temp, STBIR__CONSTI(STBIR_topscale) ); \
8764 stbir__simdi_16madd( i, i, temp ); \
8765 stbir__simdi_32shr( i, i, 16 ); \
8766}
8767
8768#define stbir__simdi_table_lookup2( v0,v1, table ) \
8769{ \
8770 stbir__simdi_u32 temp0,temp1; \
8771 temp0.m128i_i128 = v0; \
8772 temp1.m128i_i128 = v1; \
8773 temp0.m128i_u32[0] = table[temp0.m128i_i32[0]]; temp0.m128i_u32[1] = table[temp0.m128i_i32[1]]; temp0.m128i_u32[2] = table[temp0.m128i_i32[2]]; temp0.m128i_u32[3] = table[temp0.m128i_i32[3]]; \
8774 temp1.m128i_u32[0] = table[temp1.m128i_i32[0]]; temp1.m128i_u32[1] = table[temp1.m128i_i32[1]]; temp1.m128i_u32[2] = table[temp1.m128i_i32[2]]; temp1.m128i_u32[3] = table[temp1.m128i_i32[3]]; \
8775 v0 = temp0.m128i_i128; \
8776 v1 = temp1.m128i_i128; \
8777}
8778
8779#define stbir__simdi_table_lookup3( v0,v1,v2, table ) \
8780{ \
8781 stbir__simdi_u32 temp0,temp1,temp2; \
8782 temp0.m128i_i128 = v0; \
8783 temp1.m128i_i128 = v1; \
8784 temp2.m128i_i128 = v2; \
8785 temp0.m128i_u32[0] = table[temp0.m128i_i32[0]]; temp0.m128i_u32[1] = table[temp0.m128i_i32[1]]; temp0.m128i_u32[2] = table[temp0.m128i_i32[2]]; temp0.m128i_u32[3] = table[temp0.m128i_i32[3]]; \
8786 temp1.m128i_u32[0] = table[temp1.m128i_i32[0]]; temp1.m128i_u32[1] = table[temp1.m128i_i32[1]]; temp1.m128i_u32[2] = table[temp1.m128i_i32[2]]; temp1.m128i_u32[3] = table[temp1.m128i_i32[3]]; \
8787 temp2.m128i_u32[0] = table[temp2.m128i_i32[0]]; temp2.m128i_u32[1] = table[temp2.m128i_i32[1]]; temp2.m128i_u32[2] = table[temp2.m128i_i32[2]]; temp2.m128i_u32[3] = table[temp2.m128i_i32[3]]; \
8788 v0 = temp0.m128i_i128; \
8789 v1 = temp1.m128i_i128; \
8790 v2 = temp2.m128i_i128; \
8791}
8792
8793#define stbir__simdi_table_lookup4( v0,v1,v2,v3, table ) \
8794{ \
8795 stbir__simdi_u32 temp0,temp1,temp2,temp3; \
8796 temp0.m128i_i128 = v0; \
8797 temp1.m128i_i128 = v1; \
8798 temp2.m128i_i128 = v2; \
8799 temp3.m128i_i128 = v3; \
8800 temp0.m128i_u32[0] = table[temp0.m128i_i32[0]]; temp0.m128i_u32[1] = table[temp0.m128i_i32[1]]; temp0.m128i_u32[2] = table[temp0.m128i_i32[2]]; temp0.m128i_u32[3] = table[temp0.m128i_i32[3]]; \
8801 temp1.m128i_u32[0] = table[temp1.m128i_i32[0]]; temp1.m128i_u32[1] = table[temp1.m128i_i32[1]]; temp1.m128i_u32[2] = table[temp1.m128i_i32[2]]; temp1.m128i_u32[3] = table[temp1.m128i_i32[3]]; \
8802 temp2.m128i_u32[0] = table[temp2.m128i_i32[0]]; temp2.m128i_u32[1] = table[temp2.m128i_i32[1]]; temp2.m128i_u32[2] = table[temp2.m128i_i32[2]]; temp2.m128i_u32[3] = table[temp2.m128i_i32[3]]; \
8803 temp3.m128i_u32[0] = table[temp3.m128i_i32[0]]; temp3.m128i_u32[1] = table[temp3.m128i_i32[1]]; temp3.m128i_u32[2] = table[temp3.m128i_i32[2]]; temp3.m128i_u32[3] = table[temp3.m128i_i32[3]]; \
8804 v0 = temp0.m128i_i128; \
8805 v1 = temp1.m128i_i128; \
8806 v2 = temp2.m128i_i128; \
8807 v3 = temp3.m128i_i128; \
8808}
8809
8810static void STBIR__CODER_NAME( stbir__encode_uint8_srgb )( void * outputp, int width_times_channels, float const * encode )
8811{
8812 unsigned char STBIR_SIMD_STREAMOUT_PTR( * ) output = (unsigned char*) outputp;
8813 unsigned char * end_output = ( (unsigned char*) output ) + width_times_channels;
8814
8815 #ifdef STBIR_SIMD
8816
8817 if ( width_times_channels >= 16 )
8818 {
8819 float const * end_encode_m16 = encode + width_times_channels - 16;
8820 end_output -= 16;
8821 STBIR_SIMD_NO_UNROLL_LOOP_START_INF_FOR
8822 for(;;)
8823 {
8824 stbir__simdf f0, f1, f2, f3;
8825 stbir__simdi i0, i1, i2, i3;
8826 STBIR_SIMD_NO_UNROLL(encode);
8827
8828 stbir__simdf_load4_transposed( f0, f1, f2, f3, encode );
8829
8830 stbir__min_max_shift20( i0, f0 );
8831 stbir__min_max_shift20( i1, f1 );
8832 stbir__min_max_shift20( i2, f2 );
8833 stbir__min_max_shift20( i3, f3 );
8834
8835 stbir__simdi_table_lookup4( i0, i1, i2, i3, ( fp32_to_srgb8_tab4 - (127-13)*8 ) );
8836
8837 stbir__linear_to_srgb_finish( i0, f0 );
8838 stbir__linear_to_srgb_finish( i1, f1 );
8839 stbir__linear_to_srgb_finish( i2, f2 );
8840 stbir__linear_to_srgb_finish( i3, f3 );
8841
8842 stbir__interleave_pack_and_store_16_u8( output, STBIR_strs_join1(i, ,stbir__encode_order0), STBIR_strs_join1(i, ,stbir__encode_order1), STBIR_strs_join1(i, ,stbir__encode_order2), STBIR_strs_join1(i, ,stbir__encode_order3) );
8843
8844 encode += 16;
8845 output += 16;
8846 if ( output <= end_output )
8847 continue;
8848 if ( output == ( end_output + 16 ) )
8849 break;
8850 output = end_output; // backup and do last couple
8851 encode = end_encode_m16;
8852 }
8853 return;
8854 }
8855 #endif
8856
8857 // try to do blocks of 4 when you can
8858 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
8859 output += 4;
8860 STBIR_SIMD_NO_UNROLL_LOOP_START
8861 while ( output <= end_output )
8862 {
8863 STBIR_SIMD_NO_UNROLL(encode);
8864
8865 output[0-4] = stbir__linear_to_srgb_uchar( encode[stbir__encode_order0] );
8866 output[1-4] = stbir__linear_to_srgb_uchar( encode[stbir__encode_order1] );
8867 output[2-4] = stbir__linear_to_srgb_uchar( encode[stbir__encode_order2] );
8868 output[3-4] = stbir__linear_to_srgb_uchar( encode[stbir__encode_order3] );
8869
8870 output += 4;
8871 encode += 4;
8872 }
8873 output -= 4;
8874 #endif
8875
8876 // do the remnants
8877 #if stbir__coder_min_num < 4
8878 STBIR_NO_UNROLL_LOOP_START
8879 while( output < end_output )
8880 {
8881 STBIR_NO_UNROLL(encode);
8882 output[0] = stbir__linear_to_srgb_uchar( encode[stbir__encode_order0] );
8883 #if stbir__coder_min_num >= 2
8884 output[1] = stbir__linear_to_srgb_uchar( encode[stbir__encode_order1] );
8885 #endif
8886 #if stbir__coder_min_num >= 3
8887 output[2] = stbir__linear_to_srgb_uchar( encode[stbir__encode_order2] );
8888 #endif
8889 output += stbir__coder_min_num;
8890 encode += stbir__coder_min_num;
8891 }
8892 #endif
8893}
8894
8895#if ( stbir__coder_min_num == 4 ) || ( ( stbir__coder_min_num == 1 ) && ( !defined(stbir__decode_swizzle) ) )
8896
8897static float * STBIR__CODER_NAME(stbir__decode_uint8_srgb4_linearalpha)( float * decodep, int width_times_channels, void const * inputp )
8898{
8899 float STBIR_STREAMOUT_PTR( * ) decode = decodep;
8900 float * decode_end = (float*) decode + width_times_channels;
8901 unsigned char const * input = (unsigned char const *)inputp;
8902
8903 do {
8904 decode[0] = stbir__srgb_uchar_to_linear_float[ input[stbir__decode_order0] ];
8905 decode[1] = stbir__srgb_uchar_to_linear_float[ input[stbir__decode_order1] ];
8906 decode[2] = stbir__srgb_uchar_to_linear_float[ input[stbir__decode_order2] ];
8907 decode[3] = ( (float) input[stbir__decode_order3] ) * stbir__max_uint8_as_float_inverted;
8908 input += 4;
8909 decode += 4;
8910 } while( decode < decode_end );
8911 return decode_end;
8912}
8913
8914
8915static void STBIR__CODER_NAME( stbir__encode_uint8_srgb4_linearalpha )( void * outputp, int width_times_channels, float const * encode )
8916{
8917 unsigned char STBIR_SIMD_STREAMOUT_PTR( * ) output = (unsigned char*) outputp;
8918 unsigned char * end_output = ( (unsigned char*) output ) + width_times_channels;
8919
8920 #ifdef STBIR_SIMD
8921
8922 if ( width_times_channels >= 16 )
8923 {
8924 float const * end_encode_m16 = encode + width_times_channels - 16;
8925 end_output -= 16;
8926 STBIR_SIMD_NO_UNROLL_LOOP_START_INF_FOR
8927 for(;;)
8928 {
8929 stbir__simdf f0, f1, f2, f3;
8930 stbir__simdi i0, i1, i2, i3;
8931
8932 STBIR_SIMD_NO_UNROLL(encode);
8933 stbir__simdf_load4_transposed( f0, f1, f2, f3, encode );
8934
8935 stbir__min_max_shift20( i0, f0 );
8936 stbir__min_max_shift20( i1, f1 );
8937 stbir__min_max_shift20( i2, f2 );
8938 stbir__scale_and_convert( i3, f3 );
8939
8940 stbir__simdi_table_lookup3( i0, i1, i2, ( fp32_to_srgb8_tab4 - (127-13)*8 ) );
8941
8942 stbir__linear_to_srgb_finish( i0, f0 );
8943 stbir__linear_to_srgb_finish( i1, f1 );
8944 stbir__linear_to_srgb_finish( i2, f2 );
8945
8946 stbir__interleave_pack_and_store_16_u8( output, STBIR_strs_join1(i, ,stbir__encode_order0), STBIR_strs_join1(i, ,stbir__encode_order1), STBIR_strs_join1(i, ,stbir__encode_order2), STBIR_strs_join1(i, ,stbir__encode_order3) );
8947
8948 output += 16;
8949 encode += 16;
8950
8951 if ( output <= end_output )
8952 continue;
8953 if ( output == ( end_output + 16 ) )
8954 break;
8955 output = end_output; // backup and do last couple
8956 encode = end_encode_m16;
8957 }
8958 return;
8959 }
8960 #endif
8961
8962 STBIR_SIMD_NO_UNROLL_LOOP_START
8963 do {
8964 float f;
8965 STBIR_SIMD_NO_UNROLL(encode);
8966
8967 output[stbir__decode_order0] = stbir__linear_to_srgb_uchar( encode[0] );
8968 output[stbir__decode_order1] = stbir__linear_to_srgb_uchar( encode[1] );
8969 output[stbir__decode_order2] = stbir__linear_to_srgb_uchar( encode[2] );
8970
8971 f = encode[3] * stbir__max_uint8_as_float + 0.5f;
8972 STBIR_CLAMP(f, 0, 255);
8973 output[stbir__decode_order3] = (unsigned char) f;
8974
8975 output += 4;
8976 encode += 4;
8977 } while( output < end_output );
8978}
8979
8980#endif
8981
8982#if ( stbir__coder_min_num == 2 ) || ( ( stbir__coder_min_num == 1 ) && ( !defined(stbir__decode_swizzle) ) )
8983
8984static float * STBIR__CODER_NAME(stbir__decode_uint8_srgb2_linearalpha)( float * decodep, int width_times_channels, void const * inputp )
8985{
8986 float STBIR_STREAMOUT_PTR( * ) decode = decodep;
8987 float * decode_end = (float*) decode + width_times_channels;
8988 unsigned char const * input = (unsigned char const *)inputp;
8989
8990 decode += 4;
8991 while( decode <= decode_end )
8992 {
8993 decode[0-4] = stbir__srgb_uchar_to_linear_float[ input[stbir__decode_order0] ];
8994 decode[1-4] = ( (float) input[stbir__decode_order1] ) * stbir__max_uint8_as_float_inverted;
8995 decode[2-4] = stbir__srgb_uchar_to_linear_float[ input[stbir__decode_order0+2] ];
8996 decode[3-4] = ( (float) input[stbir__decode_order1+2] ) * stbir__max_uint8_as_float_inverted;
8997 input += 4;
8998 decode += 4;
8999 }
9000 decode -= 4;
9001 if( decode < decode_end )
9002 {
9003 decode[0] = stbir__srgb_uchar_to_linear_float[ input[stbir__decode_order0] ];
9004 decode[1] = ( (float) input[stbir__decode_order1] ) * stbir__max_uint8_as_float_inverted;
9005 }
9006 return decode_end;
9007}
9008
9009static void STBIR__CODER_NAME( stbir__encode_uint8_srgb2_linearalpha )( void * outputp, int width_times_channels, float const * encode )
9010{
9011 unsigned char STBIR_SIMD_STREAMOUT_PTR( * ) output = (unsigned char*) outputp;
9012 unsigned char * end_output = ( (unsigned char*) output ) + width_times_channels;
9013
9014 #ifdef STBIR_SIMD
9015
9016 if ( width_times_channels >= 16 )
9017 {
9018 float const * end_encode_m16 = encode + width_times_channels - 16;
9019 end_output -= 16;
9020 STBIR_SIMD_NO_UNROLL_LOOP_START_INF_FOR
9021 for(;;)
9022 {
9023 stbir__simdf f0, f1, f2, f3;
9024 stbir__simdi i0, i1, i2, i3;
9025
9026 STBIR_SIMD_NO_UNROLL(encode);
9027 stbir__simdf_load4_transposed( f0, f1, f2, f3, encode );
9028
9029 stbir__min_max_shift20( i0, f0 );
9030 stbir__scale_and_convert( i1, f1 );
9031 stbir__min_max_shift20( i2, f2 );
9032 stbir__scale_and_convert( i3, f3 );
9033
9034 stbir__simdi_table_lookup2( i0, i2, ( fp32_to_srgb8_tab4 - (127-13)*8 ) );
9035
9036 stbir__linear_to_srgb_finish( i0, f0 );
9037 stbir__linear_to_srgb_finish( i2, f2 );
9038
9039 stbir__interleave_pack_and_store_16_u8( output, STBIR_strs_join1(i, ,stbir__encode_order0), STBIR_strs_join1(i, ,stbir__encode_order1), STBIR_strs_join1(i, ,stbir__encode_order2), STBIR_strs_join1(i, ,stbir__encode_order3) );
9040
9041 output += 16;
9042 encode += 16;
9043 if ( output <= end_output )
9044 continue;
9045 if ( output == ( end_output + 16 ) )
9046 break;
9047 output = end_output; // backup and do last couple
9048 encode = end_encode_m16;
9049 }
9050 return;
9051 }
9052 #endif
9053
9054 STBIR_SIMD_NO_UNROLL_LOOP_START
9055 do {
9056 float f;
9057 STBIR_SIMD_NO_UNROLL(encode);
9058
9059 output[stbir__decode_order0] = stbir__linear_to_srgb_uchar( encode[0] );
9060
9061 f = encode[1] * stbir__max_uint8_as_float + 0.5f;
9062 STBIR_CLAMP(f, 0, 255);
9063 output[stbir__decode_order1] = (unsigned char) f;
9064
9065 output += 2;
9066 encode += 2;
9067 } while( output < end_output );
9068}
9069
9070#endif
9071
9072static float * STBIR__CODER_NAME(stbir__decode_uint16_linear_scaled)( float * decodep, int width_times_channels, void const * inputp )
9073{
9074 float STBIR_STREAMOUT_PTR( * ) decode = decodep;
9075 float * decode_end = (float*) decode + width_times_channels;
9076 unsigned short const * input = (unsigned short const *)inputp;
9077
9078 #ifdef STBIR_SIMD
9079 unsigned short const * end_input_m8 = input + width_times_channels - 8;
9080 if ( width_times_channels >= 8 )
9081 {
9082 decode_end -= 8;
9083 STBIR_NO_UNROLL_LOOP_START_INF_FOR
9084 for(;;)
9085 {
9086 #ifdef STBIR_SIMD8
9087 stbir__simdi i; stbir__simdi8 o;
9088 stbir__simdf8 of;
9089 STBIR_NO_UNROLL(decode);
9090 stbir__simdi_load( i, input );
9091 stbir__simdi8_expand_u16_to_u32( o, i );
9092 stbir__simdi8_convert_i32_to_float( of, o );
9093 stbir__simdf8_mult( of, of, STBIR_max_uint16_as_float_inverted8);
9094 stbir__decode_simdf8_flip( of );
9095 stbir__simdf8_store( decode + 0, of );
9096 #else
9097 stbir__simdi i, o0, o1;
9098 stbir__simdf of0, of1;
9099 STBIR_NO_UNROLL(decode);
9100 stbir__simdi_load( i, input );
9101 stbir__simdi_expand_u16_to_u32( o0,o1,i );
9102 stbir__simdi_convert_i32_to_float( of0, o0 );
9103 stbir__simdi_convert_i32_to_float( of1, o1 );
9104 stbir__simdf_mult( of0, of0, STBIR__CONSTF(STBIR_max_uint16_as_float_inverted) );
9105 stbir__simdf_mult( of1, of1, STBIR__CONSTF(STBIR_max_uint16_as_float_inverted));
9106 stbir__decode_simdf4_flip( of0 );
9107 stbir__decode_simdf4_flip( of1 );
9108 stbir__simdf_store( decode + 0, of0 );
9109 stbir__simdf_store( decode + 4, of1 );
9110 #endif
9111 decode += 8;
9112 input += 8;
9113 if ( decode <= decode_end )
9114 continue;
9115 if ( decode == ( decode_end + 8 ) )
9116 break;
9117 decode = decode_end; // backup and do last couple
9118 input = end_input_m8;
9119 }
9120 return decode_end + 8;
9121 }
9122 #endif
9123
9124 // try to do blocks of 4 when you can
9125 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
9126 decode += 4;
9127 STBIR_SIMD_NO_UNROLL_LOOP_START
9128 while( decode <= decode_end )
9129 {
9130 STBIR_SIMD_NO_UNROLL(decode);
9131 decode[0-4] = ((float)(input[stbir__decode_order0])) * stbir__max_uint16_as_float_inverted;
9132 decode[1-4] = ((float)(input[stbir__decode_order1])) * stbir__max_uint16_as_float_inverted;
9133 decode[2-4] = ((float)(input[stbir__decode_order2])) * stbir__max_uint16_as_float_inverted;
9134 decode[3-4] = ((float)(input[stbir__decode_order3])) * stbir__max_uint16_as_float_inverted;
9135 decode += 4;
9136 input += 4;
9137 }
9138 decode -= 4;
9139 #endif
9140
9141 // do the remnants
9142 #if stbir__coder_min_num < 4
9143 STBIR_NO_UNROLL_LOOP_START
9144 while( decode < decode_end )
9145 {
9146 STBIR_NO_UNROLL(decode);
9147 decode[0] = ((float)(input[stbir__decode_order0])) * stbir__max_uint16_as_float_inverted;
9148 #if stbir__coder_min_num >= 2
9149 decode[1] = ((float)(input[stbir__decode_order1])) * stbir__max_uint16_as_float_inverted;
9150 #endif
9151 #if stbir__coder_min_num >= 3
9152 decode[2] = ((float)(input[stbir__decode_order2])) * stbir__max_uint16_as_float_inverted;
9153 #endif
9154 decode += stbir__coder_min_num;
9155 input += stbir__coder_min_num;
9156 }
9157 #endif
9158 return decode_end;
9159}
9160
9161
9162static void STBIR__CODER_NAME(stbir__encode_uint16_linear_scaled)( void * outputp, int width_times_channels, float const * encode )
9163{
9164 unsigned short STBIR_SIMD_STREAMOUT_PTR( * ) output = (unsigned short*) outputp;
9165 unsigned short * end_output = ( (unsigned short*) output ) + width_times_channels;
9166
9167 #ifdef STBIR_SIMD
9168 {
9169 if ( width_times_channels >= stbir__simdfX_float_count*2 )
9170 {
9171 float const * end_encode_m8 = encode + width_times_channels - stbir__simdfX_float_count*2;
9172 end_output -= stbir__simdfX_float_count*2;
9173 STBIR_SIMD_NO_UNROLL_LOOP_START_INF_FOR
9174 for(;;)
9175 {
9176 stbir__simdfX e0, e1;
9177 stbir__simdiX i;
9178 STBIR_SIMD_NO_UNROLL(encode);
9179 stbir__simdfX_madd_mem( e0, STBIR_simd_point5X, STBIR_max_uint16_as_floatX, encode );
9180 stbir__simdfX_madd_mem( e1, STBIR_simd_point5X, STBIR_max_uint16_as_floatX, encode+stbir__simdfX_float_count );
9181 stbir__encode_simdfX_unflip( e0 );
9182 stbir__encode_simdfX_unflip( e1 );
9183 stbir__simdfX_pack_to_words( i, e0, e1 );
9184 stbir__simdiX_store( output, i );
9185 encode += stbir__simdfX_float_count*2;
9186 output += stbir__simdfX_float_count*2;
9187 if ( output <= end_output )
9188 continue;
9189 if ( output == ( end_output + stbir__simdfX_float_count*2 ) )
9190 break;
9191 output = end_output; // backup and do last couple
9192 encode = end_encode_m8;
9193 }
9194 return;
9195 }
9196 }
9197
9198 // try to do blocks of 4 when you can
9199 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
9200 output += 4;
9201 STBIR_NO_UNROLL_LOOP_START
9202 while( output <= end_output )
9203 {
9204 stbir__simdf e;
9205 stbir__simdi i;
9206 STBIR_NO_UNROLL(encode);
9207 stbir__simdf_load( e, encode );
9208 stbir__simdf_madd( e, STBIR__CONSTF(STBIR_simd_point5), STBIR__CONSTF(STBIR_max_uint16_as_float), e );
9209 stbir__encode_simdf4_unflip( e );
9210 stbir__simdf_pack_to_8words( i, e, e ); // only use first 4
9211 stbir__simdi_store2( output-4, i );
9212 output += 4;
9213 encode += 4;
9214 }
9215 output -= 4;
9216 #endif
9217
9218 // do the remnants
9219 #if stbir__coder_min_num < 4
9220 STBIR_NO_UNROLL_LOOP_START
9221 while( output < end_output )
9222 {
9223 stbir__simdf e;
9224 STBIR_NO_UNROLL(encode);
9225 stbir__simdf_madd1_mem( e, STBIR__CONSTF(STBIR_simd_point5), STBIR__CONSTF(STBIR_max_uint16_as_float), encode+stbir__encode_order0 ); output[0] = stbir__simdf_convert_float_to_short( e );
9226 #if stbir__coder_min_num >= 2
9227 stbir__simdf_madd1_mem( e, STBIR__CONSTF(STBIR_simd_point5), STBIR__CONSTF(STBIR_max_uint16_as_float), encode+stbir__encode_order1 ); output[1] = stbir__simdf_convert_float_to_short( e );
9228 #endif
9229 #if stbir__coder_min_num >= 3
9230 stbir__simdf_madd1_mem( e, STBIR__CONSTF(STBIR_simd_point5), STBIR__CONSTF(STBIR_max_uint16_as_float), encode+stbir__encode_order2 ); output[2] = stbir__simdf_convert_float_to_short( e );
9231 #endif
9232 output += stbir__coder_min_num;
9233 encode += stbir__coder_min_num;
9234 }
9235 #endif
9236
9237 #else
9238
9239 // try to do blocks of 4 when you can
9240 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
9241 output += 4;
9242 STBIR_SIMD_NO_UNROLL_LOOP_START
9243 while( output <= end_output )
9244 {
9245 float f;
9246 STBIR_SIMD_NO_UNROLL(encode);
9247 f = encode[stbir__encode_order0] * stbir__max_uint16_as_float + 0.5f; STBIR_CLAMP(f, 0, 65535); output[0-4] = (unsigned short)f;
9248 f = encode[stbir__encode_order1] * stbir__max_uint16_as_float + 0.5f; STBIR_CLAMP(f, 0, 65535); output[1-4] = (unsigned short)f;
9249 f = encode[stbir__encode_order2] * stbir__max_uint16_as_float + 0.5f; STBIR_CLAMP(f, 0, 65535); output[2-4] = (unsigned short)f;
9250 f = encode[stbir__encode_order3] * stbir__max_uint16_as_float + 0.5f; STBIR_CLAMP(f, 0, 65535); output[3-4] = (unsigned short)f;
9251 output += 4;
9252 encode += 4;
9253 }
9254 output -= 4;
9255 #endif
9256
9257 // do the remnants
9258 #if stbir__coder_min_num < 4
9259 STBIR_NO_UNROLL_LOOP_START
9260 while( output < end_output )
9261 {
9262 float f;
9263 STBIR_NO_UNROLL(encode);
9264 f = encode[stbir__encode_order0] * stbir__max_uint16_as_float + 0.5f; STBIR_CLAMP(f, 0, 65535); output[0] = (unsigned short)f;
9265 #if stbir__coder_min_num >= 2
9266 f = encode[stbir__encode_order1] * stbir__max_uint16_as_float + 0.5f; STBIR_CLAMP(f, 0, 65535); output[1] = (unsigned short)f;
9267 #endif
9268 #if stbir__coder_min_num >= 3
9269 f = encode[stbir__encode_order2] * stbir__max_uint16_as_float + 0.5f; STBIR_CLAMP(f, 0, 65535); output[2] = (unsigned short)f;
9270 #endif
9271 output += stbir__coder_min_num;
9272 encode += stbir__coder_min_num;
9273 }
9274 #endif
9275 #endif
9276}
9277
9278static float * STBIR__CODER_NAME(stbir__decode_uint16_linear)( float * decodep, int width_times_channels, void const * inputp )
9279{
9280 float STBIR_STREAMOUT_PTR( * ) decode = decodep;
9281 float * decode_end = (float*) decode + width_times_channels;
9282 unsigned short const * input = (unsigned short const *)inputp;
9283
9284 #ifdef STBIR_SIMD
9285 unsigned short const * end_input_m8 = input + width_times_channels - 8;
9286 if ( width_times_channels >= 8 )
9287 {
9288 decode_end -= 8;
9289 STBIR_NO_UNROLL_LOOP_START_INF_FOR
9290 for(;;)
9291 {
9292 #ifdef STBIR_SIMD8
9293 stbir__simdi i; stbir__simdi8 o;
9294 stbir__simdf8 of;
9295 STBIR_NO_UNROLL(decode);
9296 stbir__simdi_load( i, input );
9297 stbir__simdi8_expand_u16_to_u32( o, i );
9298 stbir__simdi8_convert_i32_to_float( of, o );
9299 stbir__decode_simdf8_flip( of );
9300 stbir__simdf8_store( decode + 0, of );
9301 #else
9302 stbir__simdi i, o0, o1;
9303 stbir__simdf of0, of1;
9304 STBIR_NO_UNROLL(decode);
9305 stbir__simdi_load( i, input );
9306 stbir__simdi_expand_u16_to_u32( o0, o1, i );
9307 stbir__simdi_convert_i32_to_float( of0, o0 );
9308 stbir__simdi_convert_i32_to_float( of1, o1 );
9309 stbir__decode_simdf4_flip( of0 );
9310 stbir__decode_simdf4_flip( of1 );
9311 stbir__simdf_store( decode + 0, of0 );
9312 stbir__simdf_store( decode + 4, of1 );
9313 #endif
9314 decode += 8;
9315 input += 8;
9316 if ( decode <= decode_end )
9317 continue;
9318 if ( decode == ( decode_end + 8 ) )
9319 break;
9320 decode = decode_end; // backup and do last couple
9321 input = end_input_m8;
9322 }
9323 return decode_end + 8;
9324 }
9325 #endif
9326
9327 // try to do blocks of 4 when you can
9328 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
9329 decode += 4;
9330 STBIR_SIMD_NO_UNROLL_LOOP_START
9331 while( decode <= decode_end )
9332 {
9333 STBIR_SIMD_NO_UNROLL(decode);
9334 decode[0-4] = ((float)(input[stbir__decode_order0]));
9335 decode[1-4] = ((float)(input[stbir__decode_order1]));
9336 decode[2-4] = ((float)(input[stbir__decode_order2]));
9337 decode[3-4] = ((float)(input[stbir__decode_order3]));
9338 decode += 4;
9339 input += 4;
9340 }
9341 decode -= 4;
9342 #endif
9343
9344 // do the remnants
9345 #if stbir__coder_min_num < 4
9346 STBIR_NO_UNROLL_LOOP_START
9347 while( decode < decode_end )
9348 {
9349 STBIR_NO_UNROLL(decode);
9350 decode[0] = ((float)(input[stbir__decode_order0]));
9351 #if stbir__coder_min_num >= 2
9352 decode[1] = ((float)(input[stbir__decode_order1]));
9353 #endif
9354 #if stbir__coder_min_num >= 3
9355 decode[2] = ((float)(input[stbir__decode_order2]));
9356 #endif
9357 decode += stbir__coder_min_num;
9358 input += stbir__coder_min_num;
9359 }
9360 #endif
9361 return decode_end;
9362}
9363
9364static void STBIR__CODER_NAME(stbir__encode_uint16_linear)( void * outputp, int width_times_channels, float const * encode )
9365{
9366 unsigned short STBIR_SIMD_STREAMOUT_PTR( * ) output = (unsigned short*) outputp;
9367 unsigned short * end_output = ( (unsigned short*) output ) + width_times_channels;
9368
9369 #ifdef STBIR_SIMD
9370 {
9371 if ( width_times_channels >= stbir__simdfX_float_count*2 )
9372 {
9373 float const * end_encode_m8 = encode + width_times_channels - stbir__simdfX_float_count*2;
9374 end_output -= stbir__simdfX_float_count*2;
9375 STBIR_SIMD_NO_UNROLL_LOOP_START_INF_FOR
9376 for(;;)
9377 {
9378 stbir__simdfX e0, e1;
9379 stbir__simdiX i;
9380 STBIR_SIMD_NO_UNROLL(encode);
9381 stbir__simdfX_add_mem( e0, STBIR_simd_point5X, encode );
9382 stbir__simdfX_add_mem( e1, STBIR_simd_point5X, encode+stbir__simdfX_float_count );
9383 stbir__encode_simdfX_unflip( e0 );
9384 stbir__encode_simdfX_unflip( e1 );
9385 stbir__simdfX_pack_to_words( i, e0, e1 );
9386 stbir__simdiX_store( output, i );
9387 encode += stbir__simdfX_float_count*2;
9388 output += stbir__simdfX_float_count*2;
9389 if ( output <= end_output )
9390 continue;
9391 if ( output == ( end_output + stbir__simdfX_float_count*2 ) )
9392 break;
9393 output = end_output; // backup and do last couple
9394 encode = end_encode_m8;
9395 }
9396 return;
9397 }
9398 }
9399
9400 // try to do blocks of 4 when you can
9401 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
9402 output += 4;
9403 STBIR_NO_UNROLL_LOOP_START
9404 while( output <= end_output )
9405 {
9406 stbir__simdf e;
9407 stbir__simdi i;
9408 STBIR_NO_UNROLL(encode);
9409 stbir__simdf_load( e, encode );
9410 stbir__simdf_add( e, STBIR__CONSTF(STBIR_simd_point5), e );
9411 stbir__encode_simdf4_unflip( e );
9412 stbir__simdf_pack_to_8words( i, e, e ); // only use first 4
9413 stbir__simdi_store2( output-4, i );
9414 output += 4;
9415 encode += 4;
9416 }
9417 output -= 4;
9418 #endif
9419
9420 #else
9421
9422 // try to do blocks of 4 when you can
9423 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
9424 output += 4;
9425 STBIR_SIMD_NO_UNROLL_LOOP_START
9426 while( output <= end_output )
9427 {
9428 float f;
9429 STBIR_SIMD_NO_UNROLL(encode);
9430 f = encode[stbir__encode_order0] + 0.5f; STBIR_CLAMP(f, 0, 65535); output[0-4] = (unsigned short)f;
9431 f = encode[stbir__encode_order1] + 0.5f; STBIR_CLAMP(f, 0, 65535); output[1-4] = (unsigned short)f;
9432 f = encode[stbir__encode_order2] + 0.5f; STBIR_CLAMP(f, 0, 65535); output[2-4] = (unsigned short)f;
9433 f = encode[stbir__encode_order3] + 0.5f; STBIR_CLAMP(f, 0, 65535); output[3-4] = (unsigned short)f;
9434 output += 4;
9435 encode += 4;
9436 }
9437 output -= 4;
9438 #endif
9439
9440 #endif
9441
9442 // do the remnants
9443 #if stbir__coder_min_num < 4
9444 STBIR_NO_UNROLL_LOOP_START
9445 while( output < end_output )
9446 {
9447 float f;
9448 STBIR_NO_UNROLL(encode);
9449 f = encode[stbir__encode_order0] + 0.5f; STBIR_CLAMP(f, 0, 65535); output[0] = (unsigned short)f;
9450 #if stbir__coder_min_num >= 2
9451 f = encode[stbir__encode_order1] + 0.5f; STBIR_CLAMP(f, 0, 65535); output[1] = (unsigned short)f;
9452 #endif
9453 #if stbir__coder_min_num >= 3
9454 f = encode[stbir__encode_order2] + 0.5f; STBIR_CLAMP(f, 0, 65535); output[2] = (unsigned short)f;
9455 #endif
9456 output += stbir__coder_min_num;
9457 encode += stbir__coder_min_num;
9458 }
9459 #endif
9460}
9461
9462static float * STBIR__CODER_NAME(stbir__decode_half_float_linear)( float * decodep, int width_times_channels, void const * inputp )
9463{
9464 float STBIR_STREAMOUT_PTR( * ) decode = decodep;
9465 float * decode_end = (float*) decode + width_times_channels;
9466 stbir__FP16 const * input = (stbir__FP16 const *)inputp;
9467
9468 #ifdef STBIR_SIMD
9469 if ( width_times_channels >= 8 )
9470 {
9471 stbir__FP16 const * end_input_m8 = input + width_times_channels - 8;
9472 decode_end -= 8;
9473 STBIR_NO_UNROLL_LOOP_START_INF_FOR
9474 for(;;)
9475 {
9476 STBIR_NO_UNROLL(decode);
9477
9478 stbir__half_to_float_SIMD( decode, input );
9479 #ifdef stbir__decode_swizzle
9480 #ifdef STBIR_SIMD8
9481 {
9482 stbir__simdf8 of;
9483 stbir__simdf8_load( of, decode );
9484 stbir__decode_simdf8_flip( of );
9485 stbir__simdf8_store( decode, of );
9486 }
9487 #else
9488 {
9489 stbir__simdf of0,of1;
9490 stbir__simdf_load( of0, decode );
9491 stbir__simdf_load( of1, decode+4 );
9492 stbir__decode_simdf4_flip( of0 );
9493 stbir__decode_simdf4_flip( of1 );
9494 stbir__simdf_store( decode, of0 );
9495 stbir__simdf_store( decode+4, of1 );
9496 }
9497 #endif
9498 #endif
9499 decode += 8;
9500 input += 8;
9501 if ( decode <= decode_end )
9502 continue;
9503 if ( decode == ( decode_end + 8 ) )
9504 break;
9505 decode = decode_end; // backup and do last couple
9506 input = end_input_m8;
9507 }
9508 return decode_end + 8;
9509 }
9510 #endif
9511
9512 // try to do blocks of 4 when you can
9513 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
9514 decode += 4;
9515 STBIR_SIMD_NO_UNROLL_LOOP_START
9516 while( decode <= decode_end )
9517 {
9518 STBIR_SIMD_NO_UNROLL(decode);
9519 decode[0-4] = stbir__half_to_float(input[stbir__decode_order0]);
9520 decode[1-4] = stbir__half_to_float(input[stbir__decode_order1]);
9521 decode[2-4] = stbir__half_to_float(input[stbir__decode_order2]);
9522 decode[3-4] = stbir__half_to_float(input[stbir__decode_order3]);
9523 decode += 4;
9524 input += 4;
9525 }
9526 decode -= 4;
9527 #endif
9528
9529 // do the remnants
9530 #if stbir__coder_min_num < 4
9531 STBIR_NO_UNROLL_LOOP_START
9532 while( decode < decode_end )
9533 {
9534 STBIR_NO_UNROLL(decode);
9535 decode[0] = stbir__half_to_float(input[stbir__decode_order0]);
9536 #if stbir__coder_min_num >= 2
9537 decode[1] = stbir__half_to_float(input[stbir__decode_order1]);
9538 #endif
9539 #if stbir__coder_min_num >= 3
9540 decode[2] = stbir__half_to_float(input[stbir__decode_order2]);
9541 #endif
9542 decode += stbir__coder_min_num;
9543 input += stbir__coder_min_num;
9544 }
9545 #endif
9546 return decode_end;
9547}
9548
9549static void STBIR__CODER_NAME( stbir__encode_half_float_linear )( void * outputp, int width_times_channels, float const * encode )
9550{
9551 stbir__FP16 STBIR_SIMD_STREAMOUT_PTR( * ) output = (stbir__FP16*) outputp;
9552 stbir__FP16 * end_output = ( (stbir__FP16*) output ) + width_times_channels;
9553
9554 #ifdef STBIR_SIMD
9555 if ( width_times_channels >= 8 )
9556 {
9557 float const * end_encode_m8 = encode + width_times_channels - 8;
9558 end_output -= 8;
9559 STBIR_SIMD_NO_UNROLL_LOOP_START_INF_FOR
9560 for(;;)
9561 {
9562 STBIR_SIMD_NO_UNROLL(encode);
9563 #ifdef stbir__decode_swizzle
9564 #ifdef STBIR_SIMD8
9565 {
9566 stbir__simdf8 of;
9567 stbir__simdf8_load( of, encode );
9568 stbir__encode_simdf8_unflip( of );
9569 stbir__float_to_half_SIMD( output, (float*)&of );
9570 }
9571 #else
9572 {
9573 stbir__simdf of[2];
9574 stbir__simdf_load( of[0], encode );
9575 stbir__simdf_load( of[1], encode+4 );
9576 stbir__encode_simdf4_unflip( of[0] );
9577 stbir__encode_simdf4_unflip( of[1] );
9578 stbir__float_to_half_SIMD( output, (float*)of );
9579 }
9580 #endif
9581 #else
9582 stbir__float_to_half_SIMD( output, encode );
9583 #endif
9584 encode += 8;
9585 output += 8;
9586 if ( output <= end_output )
9587 continue;
9588 if ( output == ( end_output + 8 ) )
9589 break;
9590 output = end_output; // backup and do last couple
9591 encode = end_encode_m8;
9592 }
9593 return;
9594 }
9595 #endif
9596
9597 // try to do blocks of 4 when you can
9598 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
9599 output += 4;
9600 STBIR_SIMD_NO_UNROLL_LOOP_START
9601 while( output <= end_output )
9602 {
9603 STBIR_SIMD_NO_UNROLL(output);
9604 output[0-4] = stbir__float_to_half(encode[stbir__encode_order0]);
9605 output[1-4] = stbir__float_to_half(encode[stbir__encode_order1]);
9606 output[2-4] = stbir__float_to_half(encode[stbir__encode_order2]);
9607 output[3-4] = stbir__float_to_half(encode[stbir__encode_order3]);
9608 output += 4;
9609 encode += 4;
9610 }
9611 output -= 4;
9612 #endif
9613
9614 // do the remnants
9615 #if stbir__coder_min_num < 4
9616 STBIR_NO_UNROLL_LOOP_START
9617 while( output < end_output )
9618 {
9619 STBIR_NO_UNROLL(output);
9620 output[0] = stbir__float_to_half(encode[stbir__encode_order0]);
9621 #if stbir__coder_min_num >= 2
9622 output[1] = stbir__float_to_half(encode[stbir__encode_order1]);
9623 #endif
9624 #if stbir__coder_min_num >= 3
9625 output[2] = stbir__float_to_half(encode[stbir__encode_order2]);
9626 #endif
9627 output += stbir__coder_min_num;
9628 encode += stbir__coder_min_num;
9629 }
9630 #endif
9631}
9632
9633static float * STBIR__CODER_NAME(stbir__decode_float_linear)( float * decodep, int width_times_channels, void const * inputp )
9634{
9635 #ifdef stbir__decode_swizzle
9636 float STBIR_STREAMOUT_PTR( * ) decode = decodep;
9637 float * decode_end = (float*) decode + width_times_channels;
9638 float const * input = (float const *)inputp;
9639
9640 #ifdef STBIR_SIMD
9641 if ( width_times_channels >= 16 )
9642 {
9643 float const * end_input_m16 = input + width_times_channels - 16;
9644 decode_end -= 16;
9645 STBIR_NO_UNROLL_LOOP_START_INF_FOR
9646 for(;;)
9647 {
9648 STBIR_NO_UNROLL(decode);
9649 #ifdef stbir__decode_swizzle
9650 #ifdef STBIR_SIMD8
9651 {
9652 stbir__simdf8 of0,of1;
9653 stbir__simdf8_load( of0, input );
9654 stbir__simdf8_load( of1, input+8 );
9655 stbir__decode_simdf8_flip( of0 );
9656 stbir__decode_simdf8_flip( of1 );
9657 stbir__simdf8_store( decode, of0 );
9658 stbir__simdf8_store( decode+8, of1 );
9659 }
9660 #else
9661 {
9662 stbir__simdf of0,of1,of2,of3;
9663 stbir__simdf_load( of0, input );
9664 stbir__simdf_load( of1, input+4 );
9665 stbir__simdf_load( of2, input+8 );
9666 stbir__simdf_load( of3, input+12 );
9667 stbir__decode_simdf4_flip( of0 );
9668 stbir__decode_simdf4_flip( of1 );
9669 stbir__decode_simdf4_flip( of2 );
9670 stbir__decode_simdf4_flip( of3 );
9671 stbir__simdf_store( decode, of0 );
9672 stbir__simdf_store( decode+4, of1 );
9673 stbir__simdf_store( decode+8, of2 );
9674 stbir__simdf_store( decode+12, of3 );
9675 }
9676 #endif
9677 #endif
9678 decode += 16;
9679 input += 16;
9680 if ( decode <= decode_end )
9681 continue;
9682 if ( decode == ( decode_end + 16 ) )
9683 break;
9684 decode = decode_end; // backup and do last couple
9685 input = end_input_m16;
9686 }
9687 return decode_end + 16;
9688 }
9689 #endif
9690
9691 // try to do blocks of 4 when you can
9692 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
9693 decode += 4;
9694 STBIR_SIMD_NO_UNROLL_LOOP_START
9695 while( decode <= decode_end )
9696 {
9697 STBIR_SIMD_NO_UNROLL(decode);
9698 decode[0-4] = input[stbir__decode_order0];
9699 decode[1-4] = input[stbir__decode_order1];
9700 decode[2-4] = input[stbir__decode_order2];
9701 decode[3-4] = input[stbir__decode_order3];
9702 decode += 4;
9703 input += 4;
9704 }
9705 decode -= 4;
9706 #endif
9707
9708 // do the remnants
9709 #if stbir__coder_min_num < 4
9710 STBIR_NO_UNROLL_LOOP_START
9711 while( decode < decode_end )
9712 {
9713 STBIR_NO_UNROLL(decode);
9714 decode[0] = input[stbir__decode_order0];
9715 #if stbir__coder_min_num >= 2
9716 decode[1] = input[stbir__decode_order1];
9717 #endif
9718 #if stbir__coder_min_num >= 3
9719 decode[2] = input[stbir__decode_order2];
9720 #endif
9721 decode += stbir__coder_min_num;
9722 input += stbir__coder_min_num;
9723 }
9724 #endif
9725 return decode_end;
9726
9727 #else
9728
9729 if ( (void*)decodep != inputp )
9730 STBIR_MEMCPY( decodep, inputp, width_times_channels * sizeof( float ) );
9731
9732 return decodep + width_times_channels;
9733
9734 #endif
9735}
9736
9737static void STBIR__CODER_NAME( stbir__encode_float_linear )( void * outputp, int width_times_channels, float const * encode )
9738{
9739 #if !defined( STBIR_FLOAT_HIGH_CLAMP ) && !defined(STBIR_FLOAT_LOW_CLAMP) && !defined(stbir__decode_swizzle)
9740
9741 if ( (void*)outputp != (void*) encode )
9742 STBIR_MEMCPY( outputp, encode, width_times_channels * sizeof( float ) );
9743
9744 #else
9745
9746 float STBIR_SIMD_STREAMOUT_PTR( * ) output = (float*) outputp;
9747 float * end_output = ( (float*) output ) + width_times_channels;
9748
9749 #ifdef STBIR_FLOAT_HIGH_CLAMP
9750 #define stbir_scalar_hi_clamp( v ) if ( v > STBIR_FLOAT_HIGH_CLAMP ) v = STBIR_FLOAT_HIGH_CLAMP;
9751 #else
9752 #define stbir_scalar_hi_clamp( v )
9753 #endif
9754 #ifdef STBIR_FLOAT_LOW_CLAMP
9755 #define stbir_scalar_lo_clamp( v ) if ( v < STBIR_FLOAT_LOW_CLAMP ) v = STBIR_FLOAT_LOW_CLAMP;
9756 #else
9757 #define stbir_scalar_lo_clamp( v )
9758 #endif
9759
9760 #ifdef STBIR_SIMD
9761
9762 #ifdef STBIR_FLOAT_HIGH_CLAMP
9763 const stbir__simdfX high_clamp = stbir__simdf_frepX(STBIR_FLOAT_HIGH_CLAMP);
9764 #endif
9765 #ifdef STBIR_FLOAT_LOW_CLAMP
9766 const stbir__simdfX low_clamp = stbir__simdf_frepX(STBIR_FLOAT_LOW_CLAMP);
9767 #endif
9768
9769 if ( width_times_channels >= ( stbir__simdfX_float_count * 2 ) )
9770 {
9771 float const * end_encode_m8 = encode + width_times_channels - ( stbir__simdfX_float_count * 2 );
9772 end_output -= ( stbir__simdfX_float_count * 2 );
9773 STBIR_SIMD_NO_UNROLL_LOOP_START_INF_FOR
9774 for(;;)
9775 {
9776 stbir__simdfX e0, e1;
9777 STBIR_SIMD_NO_UNROLL(encode);
9778 stbir__simdfX_load( e0, encode );
9779 stbir__simdfX_load( e1, encode+stbir__simdfX_float_count );
9780#ifdef STBIR_FLOAT_HIGH_CLAMP
9781 stbir__simdfX_min( e0, e0, high_clamp );
9782 stbir__simdfX_min( e1, e1, high_clamp );
9783#endif
9784#ifdef STBIR_FLOAT_LOW_CLAMP
9785 stbir__simdfX_max( e0, e0, low_clamp );
9786 stbir__simdfX_max( e1, e1, low_clamp );
9787#endif
9788 stbir__encode_simdfX_unflip( e0 );
9789 stbir__encode_simdfX_unflip( e1 );
9790 stbir__simdfX_store( output, e0 );
9791 stbir__simdfX_store( output+stbir__simdfX_float_count, e1 );
9792 encode += stbir__simdfX_float_count * 2;
9793 output += stbir__simdfX_float_count * 2;
9794 if ( output <= end_output )
9795 continue;
9796 if ( output == ( end_output + ( stbir__simdfX_float_count * 2 ) ) )
9797 break;
9798 output = end_output; // backup and do last couple
9799 encode = end_encode_m8;
9800 }
9801 return;
9802 }
9803
9804 // try to do blocks of 4 when you can
9805 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
9806 output += 4;
9807 STBIR_NO_UNROLL_LOOP_START
9808 while( output <= end_output )
9809 {
9810 stbir__simdf e0;
9811 STBIR_NO_UNROLL(encode);
9812 stbir__simdf_load( e0, encode );
9813#ifdef STBIR_FLOAT_HIGH_CLAMP
9814 stbir__simdf_min( e0, e0, high_clamp );
9815#endif
9816#ifdef STBIR_FLOAT_LOW_CLAMP
9817 stbir__simdf_max( e0, e0, low_clamp );
9818#endif
9819 stbir__encode_simdf4_unflip( e0 );
9820 stbir__simdf_store( output-4, e0 );
9821 output += 4;
9822 encode += 4;
9823 }
9824 output -= 4;
9825 #endif
9826
9827 #else
9828
9829 // try to do blocks of 4 when you can
9830 #if stbir__coder_min_num != 3 // doesn't divide cleanly by four
9831 output += 4;
9832 STBIR_SIMD_NO_UNROLL_LOOP_START
9833 while( output <= end_output )
9834 {
9835 float e;
9836 STBIR_SIMD_NO_UNROLL(encode);
9837 e = encode[ stbir__encode_order0 ]; stbir_scalar_hi_clamp( e ); stbir_scalar_lo_clamp( e ); output[0-4] = e;
9838 e = encode[ stbir__encode_order1 ]; stbir_scalar_hi_clamp( e ); stbir_scalar_lo_clamp( e ); output[1-4] = e;
9839 e = encode[ stbir__encode_order2 ]; stbir_scalar_hi_clamp( e ); stbir_scalar_lo_clamp( e ); output[2-4] = e;
9840 e = encode[ stbir__encode_order3 ]; stbir_scalar_hi_clamp( e ); stbir_scalar_lo_clamp( e ); output[3-4] = e;
9841 output += 4;
9842 encode += 4;
9843 }
9844 output -= 4;
9845
9846 #endif
9847
9848 #endif
9849
9850 // do the remnants
9851 #if stbir__coder_min_num < 4
9852 STBIR_NO_UNROLL_LOOP_START
9853 while( output < end_output )
9854 {
9855 float e;
9856 STBIR_NO_UNROLL(encode);
9857 e = encode[ stbir__encode_order0 ]; stbir_scalar_hi_clamp( e ); stbir_scalar_lo_clamp( e ); output[0] = e;
9858 #if stbir__coder_min_num >= 2
9859 e = encode[ stbir__encode_order1 ]; stbir_scalar_hi_clamp( e ); stbir_scalar_lo_clamp( e ); output[1] = e;
9860 #endif
9861 #if stbir__coder_min_num >= 3
9862 e = encode[ stbir__encode_order2 ]; stbir_scalar_hi_clamp( e ); stbir_scalar_lo_clamp( e ); output[2] = e;
9863 #endif
9864 output += stbir__coder_min_num;
9865 encode += stbir__coder_min_num;
9866 }
9867 #endif
9868
9869 #endif
9870}
9871
9872#undef stbir__decode_suffix
9873#undef stbir__decode_simdf8_flip
9874#undef stbir__decode_simdf4_flip
9875#undef stbir__decode_order0
9876#undef stbir__decode_order1
9877#undef stbir__decode_order2
9878#undef stbir__decode_order3
9879#undef stbir__encode_order0
9880#undef stbir__encode_order1
9881#undef stbir__encode_order2
9882#undef stbir__encode_order3
9883#undef stbir__encode_simdf8_unflip
9884#undef stbir__encode_simdf4_unflip
9885#undef stbir__encode_simdfX_unflip
9886#undef STBIR__CODER_NAME
9887#undef stbir__coder_min_num
9888#undef stbir__decode_swizzle
9889#undef stbir_scalar_hi_clamp
9890#undef stbir_scalar_lo_clamp
9891#undef STB_IMAGE_RESIZE_DO_CODERS
9892
9893#elif defined( STB_IMAGE_RESIZE_DO_VERTICALS)
9894
9895#ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
9896#define STBIR_chans( start, end ) STBIR_strs_join14(start,STBIR__vertical_channels,end,_cont)
9897#else
9898#define STBIR_chans( start, end ) STBIR_strs_join1(start,STBIR__vertical_channels,end)
9899#endif
9900
9901#if STBIR__vertical_channels >= 1
9902#define stbIF0( code ) code
9903#else
9904#define stbIF0( code )
9905#endif
9906#if STBIR__vertical_channels >= 2
9907#define stbIF1( code ) code
9908#else
9909#define stbIF1( code )
9910#endif
9911#if STBIR__vertical_channels >= 3
9912#define stbIF2( code ) code
9913#else
9914#define stbIF2( code )
9915#endif
9916#if STBIR__vertical_channels >= 4
9917#define stbIF3( code ) code
9918#else
9919#define stbIF3( code )
9920#endif
9921#if STBIR__vertical_channels >= 5
9922#define stbIF4( code ) code
9923#else
9924#define stbIF4( code )
9925#endif
9926#if STBIR__vertical_channels >= 6
9927#define stbIF5( code ) code
9928#else
9929#define stbIF5( code )
9930#endif
9931#if STBIR__vertical_channels >= 7
9932#define stbIF6( code ) code
9933#else
9934#define stbIF6( code )
9935#endif
9936#if STBIR__vertical_channels >= 8
9937#define stbIF7( code ) code
9938#else
9939#define stbIF7( code )
9940#endif
9941
9942static void STBIR_chans( stbir__vertical_scatter_with_,_coeffs)( float ** outputs, float const * vertical_coefficients, float const * input, float const * input_end )
9943{
9944 stbIF0( float STBIR_SIMD_STREAMOUT_PTR( * ) output0 = outputs[0]; float c0s = vertical_coefficients[0]; )
9945 stbIF1( float STBIR_SIMD_STREAMOUT_PTR( * ) output1 = outputs[1]; float c1s = vertical_coefficients[1]; )
9946 stbIF2( float STBIR_SIMD_STREAMOUT_PTR( * ) output2 = outputs[2]; float c2s = vertical_coefficients[2]; )
9947 stbIF3( float STBIR_SIMD_STREAMOUT_PTR( * ) output3 = outputs[3]; float c3s = vertical_coefficients[3]; )
9948 stbIF4( float STBIR_SIMD_STREAMOUT_PTR( * ) output4 = outputs[4]; float c4s = vertical_coefficients[4]; )
9949 stbIF5( float STBIR_SIMD_STREAMOUT_PTR( * ) output5 = outputs[5]; float c5s = vertical_coefficients[5]; )
9950 stbIF6( float STBIR_SIMD_STREAMOUT_PTR( * ) output6 = outputs[6]; float c6s = vertical_coefficients[6]; )
9951 stbIF7( float STBIR_SIMD_STREAMOUT_PTR( * ) output7 = outputs[7]; float c7s = vertical_coefficients[7]; )
9952
9953 #ifdef STBIR_SIMD
9954 {
9955 stbIF0(stbir__simdfX c0 = stbir__simdf_frepX( c0s ); )
9956 stbIF1(stbir__simdfX c1 = stbir__simdf_frepX( c1s ); )
9957 stbIF2(stbir__simdfX c2 = stbir__simdf_frepX( c2s ); )
9958 stbIF3(stbir__simdfX c3 = stbir__simdf_frepX( c3s ); )
9959 stbIF4(stbir__simdfX c4 = stbir__simdf_frepX( c4s ); )
9960 stbIF5(stbir__simdfX c5 = stbir__simdf_frepX( c5s ); )
9961 stbIF6(stbir__simdfX c6 = stbir__simdf_frepX( c6s ); )
9962 stbIF7(stbir__simdfX c7 = stbir__simdf_frepX( c7s ); )
9963 STBIR_SIMD_NO_UNROLL_LOOP_START
9964 while ( ( (char*)input_end - (char*) input ) >= (16*stbir__simdfX_float_count) )
9965 {
9966 stbir__simdfX o0, o1, o2, o3, r0, r1, r2, r3;
9967 STBIR_SIMD_NO_UNROLL(output0);
9968
9969 stbir__simdfX_load( r0, input ); stbir__simdfX_load( r1, input+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input+(3*stbir__simdfX_float_count) );
9970
9971 #ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
9972 stbIF0( stbir__simdfX_load( o0, output0 ); stbir__simdfX_load( o1, output0+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output0+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( o3, output0+(3*stbir__simdfX_float_count) );
9973 stbir__simdfX_madd( o0, o0, r0, c0 ); stbir__simdfX_madd( o1, o1, r1, c0 ); stbir__simdfX_madd( o2, o2, r2, c0 ); stbir__simdfX_madd( o3, o3, r3, c0 );
9974 stbir__simdfX_store( output0, o0 ); stbir__simdfX_store( output0+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output0+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output0+(3*stbir__simdfX_float_count), o3 ); )
9975 stbIF1( stbir__simdfX_load( o0, output1 ); stbir__simdfX_load( o1, output1+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output1+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( o3, output1+(3*stbir__simdfX_float_count) );
9976 stbir__simdfX_madd( o0, o0, r0, c1 ); stbir__simdfX_madd( o1, o1, r1, c1 ); stbir__simdfX_madd( o2, o2, r2, c1 ); stbir__simdfX_madd( o3, o3, r3, c1 );
9977 stbir__simdfX_store( output1, o0 ); stbir__simdfX_store( output1+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output1+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output1+(3*stbir__simdfX_float_count), o3 ); )
9978 stbIF2( stbir__simdfX_load( o0, output2 ); stbir__simdfX_load( o1, output2+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output2+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( o3, output2+(3*stbir__simdfX_float_count) );
9979 stbir__simdfX_madd( o0, o0, r0, c2 ); stbir__simdfX_madd( o1, o1, r1, c2 ); stbir__simdfX_madd( o2, o2, r2, c2 ); stbir__simdfX_madd( o3, o3, r3, c2 );
9980 stbir__simdfX_store( output2, o0 ); stbir__simdfX_store( output2+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output2+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output2+(3*stbir__simdfX_float_count), o3 ); )
9981 stbIF3( stbir__simdfX_load( o0, output3 ); stbir__simdfX_load( o1, output3+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output3+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( o3, output3+(3*stbir__simdfX_float_count) );
9982 stbir__simdfX_madd( o0, o0, r0, c3 ); stbir__simdfX_madd( o1, o1, r1, c3 ); stbir__simdfX_madd( o2, o2, r2, c3 ); stbir__simdfX_madd( o3, o3, r3, c3 );
9983 stbir__simdfX_store( output3, o0 ); stbir__simdfX_store( output3+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output3+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output3+(3*stbir__simdfX_float_count), o3 ); )
9984 stbIF4( stbir__simdfX_load( o0, output4 ); stbir__simdfX_load( o1, output4+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output4+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( o3, output4+(3*stbir__simdfX_float_count) );
9985 stbir__simdfX_madd( o0, o0, r0, c4 ); stbir__simdfX_madd( o1, o1, r1, c4 ); stbir__simdfX_madd( o2, o2, r2, c4 ); stbir__simdfX_madd( o3, o3, r3, c4 );
9986 stbir__simdfX_store( output4, o0 ); stbir__simdfX_store( output4+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output4+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output4+(3*stbir__simdfX_float_count), o3 ); )
9987 stbIF5( stbir__simdfX_load( o0, output5 ); stbir__simdfX_load( o1, output5+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output5+(2*stbir__simdfX_float_count)); stbir__simdfX_load( o3, output5+(3*stbir__simdfX_float_count) );
9988 stbir__simdfX_madd( o0, o0, r0, c5 ); stbir__simdfX_madd( o1, o1, r1, c5 ); stbir__simdfX_madd( o2, o2, r2, c5 ); stbir__simdfX_madd( o3, o3, r3, c5 );
9989 stbir__simdfX_store( output5, o0 ); stbir__simdfX_store( output5+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output5+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output5+(3*stbir__simdfX_float_count), o3 ); )
9990 stbIF6( stbir__simdfX_load( o0, output6 ); stbir__simdfX_load( o1, output6+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output6+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( o3, output6+(3*stbir__simdfX_float_count) );
9991 stbir__simdfX_madd( o0, o0, r0, c6 ); stbir__simdfX_madd( o1, o1, r1, c6 ); stbir__simdfX_madd( o2, o2, r2, c6 ); stbir__simdfX_madd( o3, o3, r3, c6 );
9992 stbir__simdfX_store( output6, o0 ); stbir__simdfX_store( output6+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output6+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output6+(3*stbir__simdfX_float_count), o3 ); )
9993 stbIF7( stbir__simdfX_load( o0, output7 ); stbir__simdfX_load( o1, output7+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output7+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( o3, output7+(3*stbir__simdfX_float_count) );
9994 stbir__simdfX_madd( o0, o0, r0, c7 ); stbir__simdfX_madd( o1, o1, r1, c7 ); stbir__simdfX_madd( o2, o2, r2, c7 ); stbir__simdfX_madd( o3, o3, r3, c7 );
9995 stbir__simdfX_store( output7, o0 ); stbir__simdfX_store( output7+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output7+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output7+(3*stbir__simdfX_float_count), o3 ); )
9996 #else
9997 stbIF0( stbir__simdfX_mult( o0, r0, c0 ); stbir__simdfX_mult( o1, r1, c0 ); stbir__simdfX_mult( o2, r2, c0 ); stbir__simdfX_mult( o3, r3, c0 );
9998 stbir__simdfX_store( output0, o0 ); stbir__simdfX_store( output0+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output0+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output0+(3*stbir__simdfX_float_count), o3 ); )
9999 stbIF1( stbir__simdfX_mult( o0, r0, c1 ); stbir__simdfX_mult( o1, r1, c1 ); stbir__simdfX_mult( o2, r2, c1 ); stbir__simdfX_mult( o3, r3, c1 );
10000 stbir__simdfX_store( output1, o0 ); stbir__simdfX_store( output1+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output1+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output1+(3*stbir__simdfX_float_count), o3 ); )
10001 stbIF2( stbir__simdfX_mult( o0, r0, c2 ); stbir__simdfX_mult( o1, r1, c2 ); stbir__simdfX_mult( o2, r2, c2 ); stbir__simdfX_mult( o3, r3, c2 );
10002 stbir__simdfX_store( output2, o0 ); stbir__simdfX_store( output2+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output2+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output2+(3*stbir__simdfX_float_count), o3 ); )
10003 stbIF3( stbir__simdfX_mult( o0, r0, c3 ); stbir__simdfX_mult( o1, r1, c3 ); stbir__simdfX_mult( o2, r2, c3 ); stbir__simdfX_mult( o3, r3, c3 );
10004 stbir__simdfX_store( output3, o0 ); stbir__simdfX_store( output3+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output3+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output3+(3*stbir__simdfX_float_count), o3 ); )
10005 stbIF4( stbir__simdfX_mult( o0, r0, c4 ); stbir__simdfX_mult( o1, r1, c4 ); stbir__simdfX_mult( o2, r2, c4 ); stbir__simdfX_mult( o3, r3, c4 );
10006 stbir__simdfX_store( output4, o0 ); stbir__simdfX_store( output4+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output4+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output4+(3*stbir__simdfX_float_count), o3 ); )
10007 stbIF5( stbir__simdfX_mult( o0, r0, c5 ); stbir__simdfX_mult( o1, r1, c5 ); stbir__simdfX_mult( o2, r2, c5 ); stbir__simdfX_mult( o3, r3, c5 );
10008 stbir__simdfX_store( output5, o0 ); stbir__simdfX_store( output5+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output5+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output5+(3*stbir__simdfX_float_count), o3 ); )
10009 stbIF6( stbir__simdfX_mult( o0, r0, c6 ); stbir__simdfX_mult( o1, r1, c6 ); stbir__simdfX_mult( o2, r2, c6 ); stbir__simdfX_mult( o3, r3, c6 );
10010 stbir__simdfX_store( output6, o0 ); stbir__simdfX_store( output6+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output6+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output6+(3*stbir__simdfX_float_count), o3 ); )
10011 stbIF7( stbir__simdfX_mult( o0, r0, c7 ); stbir__simdfX_mult( o1, r1, c7 ); stbir__simdfX_mult( o2, r2, c7 ); stbir__simdfX_mult( o3, r3, c7 );
10012 stbir__simdfX_store( output7, o0 ); stbir__simdfX_store( output7+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output7+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output7+(3*stbir__simdfX_float_count), o3 ); )
10013 #endif
10014
10015 input += (4*stbir__simdfX_float_count);
10016 stbIF0( output0 += (4*stbir__simdfX_float_count); ) stbIF1( output1 += (4*stbir__simdfX_float_count); ) stbIF2( output2 += (4*stbir__simdfX_float_count); ) stbIF3( output3 += (4*stbir__simdfX_float_count); ) stbIF4( output4 += (4*stbir__simdfX_float_count); ) stbIF5( output5 += (4*stbir__simdfX_float_count); ) stbIF6( output6 += (4*stbir__simdfX_float_count); ) stbIF7( output7 += (4*stbir__simdfX_float_count); )
10017 }
10018 STBIR_SIMD_NO_UNROLL_LOOP_START
10019 while ( ( (char*)input_end - (char*) input ) >= 16 )
10020 {
10021 stbir__simdf o0, r0;
10022 STBIR_SIMD_NO_UNROLL(output0);
10023
10024 stbir__simdf_load( r0, input );
10025
10026 #ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
10027 stbIF0( stbir__simdf_load( o0, output0 ); stbir__simdf_madd( o0, o0, r0, stbir__if_simdf8_cast_to_simdf4( c0 ) ); stbir__simdf_store( output0, o0 ); )
10028 stbIF1( stbir__simdf_load( o0, output1 ); stbir__simdf_madd( o0, o0, r0, stbir__if_simdf8_cast_to_simdf4( c1 ) ); stbir__simdf_store( output1, o0 ); )
10029 stbIF2( stbir__simdf_load( o0, output2 ); stbir__simdf_madd( o0, o0, r0, stbir__if_simdf8_cast_to_simdf4( c2 ) ); stbir__simdf_store( output2, o0 ); )
10030 stbIF3( stbir__simdf_load( o0, output3 ); stbir__simdf_madd( o0, o0, r0, stbir__if_simdf8_cast_to_simdf4( c3 ) ); stbir__simdf_store( output3, o0 ); )
10031 stbIF4( stbir__simdf_load( o0, output4 ); stbir__simdf_madd( o0, o0, r0, stbir__if_simdf8_cast_to_simdf4( c4 ) ); stbir__simdf_store( output4, o0 ); )
10032 stbIF5( stbir__simdf_load( o0, output5 ); stbir__simdf_madd( o0, o0, r0, stbir__if_simdf8_cast_to_simdf4( c5 ) ); stbir__simdf_store( output5, o0 ); )
10033 stbIF6( stbir__simdf_load( o0, output6 ); stbir__simdf_madd( o0, o0, r0, stbir__if_simdf8_cast_to_simdf4( c6 ) ); stbir__simdf_store( output6, o0 ); )
10034 stbIF7( stbir__simdf_load( o0, output7 ); stbir__simdf_madd( o0, o0, r0, stbir__if_simdf8_cast_to_simdf4( c7 ) ); stbir__simdf_store( output7, o0 ); )
10035 #else
10036 stbIF0( stbir__simdf_mult( o0, r0, stbir__if_simdf8_cast_to_simdf4( c0 ) ); stbir__simdf_store( output0, o0 ); )
10037 stbIF1( stbir__simdf_mult( o0, r0, stbir__if_simdf8_cast_to_simdf4( c1 ) ); stbir__simdf_store( output1, o0 ); )
10038 stbIF2( stbir__simdf_mult( o0, r0, stbir__if_simdf8_cast_to_simdf4( c2 ) ); stbir__simdf_store( output2, o0 ); )
10039 stbIF3( stbir__simdf_mult( o0, r0, stbir__if_simdf8_cast_to_simdf4( c3 ) ); stbir__simdf_store( output3, o0 ); )
10040 stbIF4( stbir__simdf_mult( o0, r0, stbir__if_simdf8_cast_to_simdf4( c4 ) ); stbir__simdf_store( output4, o0 ); )
10041 stbIF5( stbir__simdf_mult( o0, r0, stbir__if_simdf8_cast_to_simdf4( c5 ) ); stbir__simdf_store( output5, o0 ); )
10042 stbIF6( stbir__simdf_mult( o0, r0, stbir__if_simdf8_cast_to_simdf4( c6 ) ); stbir__simdf_store( output6, o0 ); )
10043 stbIF7( stbir__simdf_mult( o0, r0, stbir__if_simdf8_cast_to_simdf4( c7 ) ); stbir__simdf_store( output7, o0 ); )
10044 #endif
10045
10046 input += 4;
10047 stbIF0( output0 += 4; ) stbIF1( output1 += 4; ) stbIF2( output2 += 4; ) stbIF3( output3 += 4; ) stbIF4( output4 += 4; ) stbIF5( output5 += 4; ) stbIF6( output6 += 4; ) stbIF7( output7 += 4; )
10048 }
10049 }
10050 #else
10051 STBIR_NO_UNROLL_LOOP_START
10052 while ( ( (char*)input_end - (char*) input ) >= 16 )
10053 {
10054 float r0, r1, r2, r3;
10055 STBIR_NO_UNROLL(input);
10056
10057 r0 = input[0], r1 = input[1], r2 = input[2], r3 = input[3];
10058
10059 #ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
10060 stbIF0( output0[0] += ( r0 * c0s ); output0[1] += ( r1 * c0s ); output0[2] += ( r2 * c0s ); output0[3] += ( r3 * c0s ); )
10061 stbIF1( output1[0] += ( r0 * c1s ); output1[1] += ( r1 * c1s ); output1[2] += ( r2 * c1s ); output1[3] += ( r3 * c1s ); )
10062 stbIF2( output2[0] += ( r0 * c2s ); output2[1] += ( r1 * c2s ); output2[2] += ( r2 * c2s ); output2[3] += ( r3 * c2s ); )
10063 stbIF3( output3[0] += ( r0 * c3s ); output3[1] += ( r1 * c3s ); output3[2] += ( r2 * c3s ); output3[3] += ( r3 * c3s ); )
10064 stbIF4( output4[0] += ( r0 * c4s ); output4[1] += ( r1 * c4s ); output4[2] += ( r2 * c4s ); output4[3] += ( r3 * c4s ); )
10065 stbIF5( output5[0] += ( r0 * c5s ); output5[1] += ( r1 * c5s ); output5[2] += ( r2 * c5s ); output5[3] += ( r3 * c5s ); )
10066 stbIF6( output6[0] += ( r0 * c6s ); output6[1] += ( r1 * c6s ); output6[2] += ( r2 * c6s ); output6[3] += ( r3 * c6s ); )
10067 stbIF7( output7[0] += ( r0 * c7s ); output7[1] += ( r1 * c7s ); output7[2] += ( r2 * c7s ); output7[3] += ( r3 * c7s ); )
10068 #else
10069 stbIF0( output0[0] = ( r0 * c0s ); output0[1] = ( r1 * c0s ); output0[2] = ( r2 * c0s ); output0[3] = ( r3 * c0s ); )
10070 stbIF1( output1[0] = ( r0 * c1s ); output1[1] = ( r1 * c1s ); output1[2] = ( r2 * c1s ); output1[3] = ( r3 * c1s ); )
10071 stbIF2( output2[0] = ( r0 * c2s ); output2[1] = ( r1 * c2s ); output2[2] = ( r2 * c2s ); output2[3] = ( r3 * c2s ); )
10072 stbIF3( output3[0] = ( r0 * c3s ); output3[1] = ( r1 * c3s ); output3[2] = ( r2 * c3s ); output3[3] = ( r3 * c3s ); )
10073 stbIF4( output4[0] = ( r0 * c4s ); output4[1] = ( r1 * c4s ); output4[2] = ( r2 * c4s ); output4[3] = ( r3 * c4s ); )
10074 stbIF5( output5[0] = ( r0 * c5s ); output5[1] = ( r1 * c5s ); output5[2] = ( r2 * c5s ); output5[3] = ( r3 * c5s ); )
10075 stbIF6( output6[0] = ( r0 * c6s ); output6[1] = ( r1 * c6s ); output6[2] = ( r2 * c6s ); output6[3] = ( r3 * c6s ); )
10076 stbIF7( output7[0] = ( r0 * c7s ); output7[1] = ( r1 * c7s ); output7[2] = ( r2 * c7s ); output7[3] = ( r3 * c7s ); )
10077 #endif
10078
10079 input += 4;
10080 stbIF0( output0 += 4; ) stbIF1( output1 += 4; ) stbIF2( output2 += 4; ) stbIF3( output3 += 4; ) stbIF4( output4 += 4; ) stbIF5( output5 += 4; ) stbIF6( output6 += 4; ) stbIF7( output7 += 4; )
10081 }
10082 #endif
10083 STBIR_NO_UNROLL_LOOP_START
10084 while ( input < input_end )
10085 {
10086 float r = input[0];
10087 STBIR_NO_UNROLL(output0);
10088
10089 #ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
10090 stbIF0( output0[0] += ( r * c0s ); )
10091 stbIF1( output1[0] += ( r * c1s ); )
10092 stbIF2( output2[0] += ( r * c2s ); )
10093 stbIF3( output3[0] += ( r * c3s ); )
10094 stbIF4( output4[0] += ( r * c4s ); )
10095 stbIF5( output5[0] += ( r * c5s ); )
10096 stbIF6( output6[0] += ( r * c6s ); )
10097 stbIF7( output7[0] += ( r * c7s ); )
10098 #else
10099 stbIF0( output0[0] = ( r * c0s ); )
10100 stbIF1( output1[0] = ( r * c1s ); )
10101 stbIF2( output2[0] = ( r * c2s ); )
10102 stbIF3( output3[0] = ( r * c3s ); )
10103 stbIF4( output4[0] = ( r * c4s ); )
10104 stbIF5( output5[0] = ( r * c5s ); )
10105 stbIF6( output6[0] = ( r * c6s ); )
10106 stbIF7( output7[0] = ( r * c7s ); )
10107 #endif
10108
10109 ++input;
10110 stbIF0( ++output0; ) stbIF1( ++output1; ) stbIF2( ++output2; ) stbIF3( ++output3; ) stbIF4( ++output4; ) stbIF5( ++output5; ) stbIF6( ++output6; ) stbIF7( ++output7; )
10111 }
10112}
10113
10114static void STBIR_chans( stbir__vertical_gather_with_,_coeffs)( float * outputp, float const * vertical_coefficients, float const ** inputs, float const * input0_end )
10115{
10116 float STBIR_SIMD_STREAMOUT_PTR( * ) output = outputp;
10117
10118 stbIF0( float const * input0 = inputs[0]; float c0s = vertical_coefficients[0]; )
10119 stbIF1( float const * input1 = inputs[1]; float c1s = vertical_coefficients[1]; )
10120 stbIF2( float const * input2 = inputs[2]; float c2s = vertical_coefficients[2]; )
10121 stbIF3( float const * input3 = inputs[3]; float c3s = vertical_coefficients[3]; )
10122 stbIF4( float const * input4 = inputs[4]; float c4s = vertical_coefficients[4]; )
10123 stbIF5( float const * input5 = inputs[5]; float c5s = vertical_coefficients[5]; )
10124 stbIF6( float const * input6 = inputs[6]; float c6s = vertical_coefficients[6]; )
10125 stbIF7( float const * input7 = inputs[7]; float c7s = vertical_coefficients[7]; )
10126
10127#if ( STBIR__vertical_channels == 1 ) && !defined(STB_IMAGE_RESIZE_VERTICAL_CONTINUE)
10128 // check single channel one weight
10129 if ( ( c0s >= (1.0f-0.000001f) ) && ( c0s <= (1.0f+0.000001f) ) )
10130 {
10131 STBIR_MEMCPY( output, input0, (char*)input0_end - (char*)input0 );
10132 return;
10133 }
10134#endif
10135
10136 #ifdef STBIR_SIMD
10137 {
10138 stbIF0(stbir__simdfX c0 = stbir__simdf_frepX( c0s ); )
10139 stbIF1(stbir__simdfX c1 = stbir__simdf_frepX( c1s ); )
10140 stbIF2(stbir__simdfX c2 = stbir__simdf_frepX( c2s ); )
10141 stbIF3(stbir__simdfX c3 = stbir__simdf_frepX( c3s ); )
10142 stbIF4(stbir__simdfX c4 = stbir__simdf_frepX( c4s ); )
10143 stbIF5(stbir__simdfX c5 = stbir__simdf_frepX( c5s ); )
10144 stbIF6(stbir__simdfX c6 = stbir__simdf_frepX( c6s ); )
10145 stbIF7(stbir__simdfX c7 = stbir__simdf_frepX( c7s ); )
10146
10147 STBIR_SIMD_NO_UNROLL_LOOP_START
10148 while ( ( (char*)input0_end - (char*) input0 ) >= (16*stbir__simdfX_float_count) )
10149 {
10150 stbir__simdfX o0, o1, o2, o3, r0, r1, r2, r3;
10151 STBIR_SIMD_NO_UNROLL(output);
10152
10153 // prefetch four loop iterations ahead (doesn't affect much for small resizes, but helps with big ones)
10154 stbIF0( stbir__prefetch( input0 + (16*stbir__simdfX_float_count) ); )
10155 stbIF1( stbir__prefetch( input1 + (16*stbir__simdfX_float_count) ); )
10156 stbIF2( stbir__prefetch( input2 + (16*stbir__simdfX_float_count) ); )
10157 stbIF3( stbir__prefetch( input3 + (16*stbir__simdfX_float_count) ); )
10158 stbIF4( stbir__prefetch( input4 + (16*stbir__simdfX_float_count) ); )
10159 stbIF5( stbir__prefetch( input5 + (16*stbir__simdfX_float_count) ); )
10160 stbIF6( stbir__prefetch( input6 + (16*stbir__simdfX_float_count) ); )
10161 stbIF7( stbir__prefetch( input7 + (16*stbir__simdfX_float_count) ); )
10162
10163 #ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
10164 stbIF0( stbir__simdfX_load( o0, output ); stbir__simdfX_load( o1, output+stbir__simdfX_float_count ); stbir__simdfX_load( o2, output+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( o3, output+(3*stbir__simdfX_float_count) );
10165 stbir__simdfX_load( r0, input0 ); stbir__simdfX_load( r1, input0+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input0+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input0+(3*stbir__simdfX_float_count) );
10166 stbir__simdfX_madd( o0, o0, r0, c0 ); stbir__simdfX_madd( o1, o1, r1, c0 ); stbir__simdfX_madd( o2, o2, r2, c0 ); stbir__simdfX_madd( o3, o3, r3, c0 ); )
10167 #else
10168 stbIF0( stbir__simdfX_load( r0, input0 ); stbir__simdfX_load( r1, input0+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input0+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input0+(3*stbir__simdfX_float_count) );
10169 stbir__simdfX_mult( o0, r0, c0 ); stbir__simdfX_mult( o1, r1, c0 ); stbir__simdfX_mult( o2, r2, c0 ); stbir__simdfX_mult( o3, r3, c0 ); )
10170 #endif
10171
10172 stbIF1( stbir__simdfX_load( r0, input1 ); stbir__simdfX_load( r1, input1+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input1+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input1+(3*stbir__simdfX_float_count) );
10173 stbir__simdfX_madd( o0, o0, r0, c1 ); stbir__simdfX_madd( o1, o1, r1, c1 ); stbir__simdfX_madd( o2, o2, r2, c1 ); stbir__simdfX_madd( o3, o3, r3, c1 ); )
10174 stbIF2( stbir__simdfX_load( r0, input2 ); stbir__simdfX_load( r1, input2+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input2+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input2+(3*stbir__simdfX_float_count) );
10175 stbir__simdfX_madd( o0, o0, r0, c2 ); stbir__simdfX_madd( o1, o1, r1, c2 ); stbir__simdfX_madd( o2, o2, r2, c2 ); stbir__simdfX_madd( o3, o3, r3, c2 ); )
10176 stbIF3( stbir__simdfX_load( r0, input3 ); stbir__simdfX_load( r1, input3+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input3+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input3+(3*stbir__simdfX_float_count) );
10177 stbir__simdfX_madd( o0, o0, r0, c3 ); stbir__simdfX_madd( o1, o1, r1, c3 ); stbir__simdfX_madd( o2, o2, r2, c3 ); stbir__simdfX_madd( o3, o3, r3, c3 ); )
10178 stbIF4( stbir__simdfX_load( r0, input4 ); stbir__simdfX_load( r1, input4+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input4+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input4+(3*stbir__simdfX_float_count) );
10179 stbir__simdfX_madd( o0, o0, r0, c4 ); stbir__simdfX_madd( o1, o1, r1, c4 ); stbir__simdfX_madd( o2, o2, r2, c4 ); stbir__simdfX_madd( o3, o3, r3, c4 ); )
10180 stbIF5( stbir__simdfX_load( r0, input5 ); stbir__simdfX_load( r1, input5+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input5+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input5+(3*stbir__simdfX_float_count) );
10181 stbir__simdfX_madd( o0, o0, r0, c5 ); stbir__simdfX_madd( o1, o1, r1, c5 ); stbir__simdfX_madd( o2, o2, r2, c5 ); stbir__simdfX_madd( o3, o3, r3, c5 ); )
10182 stbIF6( stbir__simdfX_load( r0, input6 ); stbir__simdfX_load( r1, input6+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input6+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input6+(3*stbir__simdfX_float_count) );
10183 stbir__simdfX_madd( o0, o0, r0, c6 ); stbir__simdfX_madd( o1, o1, r1, c6 ); stbir__simdfX_madd( o2, o2, r2, c6 ); stbir__simdfX_madd( o3, o3, r3, c6 ); )
10184 stbIF7( stbir__simdfX_load( r0, input7 ); stbir__simdfX_load( r1, input7+stbir__simdfX_float_count ); stbir__simdfX_load( r2, input7+(2*stbir__simdfX_float_count) ); stbir__simdfX_load( r3, input7+(3*stbir__simdfX_float_count) );
10185 stbir__simdfX_madd( o0, o0, r0, c7 ); stbir__simdfX_madd( o1, o1, r1, c7 ); stbir__simdfX_madd( o2, o2, r2, c7 ); stbir__simdfX_madd( o3, o3, r3, c7 ); )
10186
10187 stbir__simdfX_store( output, o0 ); stbir__simdfX_store( output+stbir__simdfX_float_count, o1 ); stbir__simdfX_store( output+(2*stbir__simdfX_float_count), o2 ); stbir__simdfX_store( output+(3*stbir__simdfX_float_count), o3 );
10188 output += (4*stbir__simdfX_float_count);
10189 stbIF0( input0 += (4*stbir__simdfX_float_count); ) stbIF1( input1 += (4*stbir__simdfX_float_count); ) stbIF2( input2 += (4*stbir__simdfX_float_count); ) stbIF3( input3 += (4*stbir__simdfX_float_count); ) stbIF4( input4 += (4*stbir__simdfX_float_count); ) stbIF5( input5 += (4*stbir__simdfX_float_count); ) stbIF6( input6 += (4*stbir__simdfX_float_count); ) stbIF7( input7 += (4*stbir__simdfX_float_count); )
10190 }
10191
10192 STBIR_SIMD_NO_UNROLL_LOOP_START
10193 while ( ( (char*)input0_end - (char*) input0 ) >= 16 )
10194 {
10195 stbir__simdf o0, r0;
10196 STBIR_SIMD_NO_UNROLL(output);
10197
10198 #ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
10199 stbIF0( stbir__simdf_load( o0, output ); stbir__simdf_load( r0, input0 ); stbir__simdf_madd( o0, o0, r0, stbir__if_simdf8_cast_to_simdf4( c0 ) ); )
10200 #else
10201 stbIF0( stbir__simdf_load( r0, input0 ); stbir__simdf_mult( o0, r0, stbir__if_simdf8_cast_to_simdf4( c0 ) ); )
10202 #endif
10203 stbIF1( stbir__simdf_load( r0, input1 ); stbir__simdf_madd( o0, o0, r0, stbir__if_simdf8_cast_to_simdf4( c1 ) ); )
10204 stbIF2( stbir__simdf_load( r0, input2 ); stbir__simdf_madd( o0, o0, r0, stbir__if_simdf8_cast_to_simdf4( c2 ) ); )
10205 stbIF3( stbir__simdf_load( r0, input3 ); stbir__simdf_madd( o0, o0, r0, stbir__if_simdf8_cast_to_simdf4( c3 ) ); )
10206 stbIF4( stbir__simdf_load( r0, input4 ); stbir__simdf_madd( o0, o0, r0, stbir__if_simdf8_cast_to_simdf4( c4 ) ); )
10207 stbIF5( stbir__simdf_load( r0, input5 ); stbir__simdf_madd( o0, o0, r0, stbir__if_simdf8_cast_to_simdf4( c5 ) ); )
10208 stbIF6( stbir__simdf_load( r0, input6 ); stbir__simdf_madd( o0, o0, r0, stbir__if_simdf8_cast_to_simdf4( c6 ) ); )
10209 stbIF7( stbir__simdf_load( r0, input7 ); stbir__simdf_madd( o0, o0, r0, stbir__if_simdf8_cast_to_simdf4( c7 ) ); )
10210
10211 stbir__simdf_store( output, o0 );
10212 output += 4;
10213 stbIF0( input0 += 4; ) stbIF1( input1 += 4; ) stbIF2( input2 += 4; ) stbIF3( input3 += 4; ) stbIF4( input4 += 4; ) stbIF5( input5 += 4; ) stbIF6( input6 += 4; ) stbIF7( input7 += 4; )
10214 }
10215 }
10216 #else
10217 STBIR_NO_UNROLL_LOOP_START
10218 while ( ( (char*)input0_end - (char*) input0 ) >= 16 )
10219 {
10220 float o0, o1, o2, o3;
10221 STBIR_NO_UNROLL(output);
10222 #ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
10223 stbIF0( o0 = output[0] + input0[0] * c0s; o1 = output[1] + input0[1] * c0s; o2 = output[2] + input0[2] * c0s; o3 = output[3] + input0[3] * c0s; )
10224 #else
10225 stbIF0( o0 = input0[0] * c0s; o1 = input0[1] * c0s; o2 = input0[2] * c0s; o3 = input0[3] * c0s; )
10226 #endif
10227 stbIF1( o0 += input1[0] * c1s; o1 += input1[1] * c1s; o2 += input1[2] * c1s; o3 += input1[3] * c1s; )
10228 stbIF2( o0 += input2[0] * c2s; o1 += input2[1] * c2s; o2 += input2[2] * c2s; o3 += input2[3] * c2s; )
10229 stbIF3( o0 += input3[0] * c3s; o1 += input3[1] * c3s; o2 += input3[2] * c3s; o3 += input3[3] * c3s; )
10230 stbIF4( o0 += input4[0] * c4s; o1 += input4[1] * c4s; o2 += input4[2] * c4s; o3 += input4[3] * c4s; )
10231 stbIF5( o0 += input5[0] * c5s; o1 += input5[1] * c5s; o2 += input5[2] * c5s; o3 += input5[3] * c5s; )
10232 stbIF6( o0 += input6[0] * c6s; o1 += input6[1] * c6s; o2 += input6[2] * c6s; o3 += input6[3] * c6s; )
10233 stbIF7( o0 += input7[0] * c7s; o1 += input7[1] * c7s; o2 += input7[2] * c7s; o3 += input7[3] * c7s; )
10234 output[0] = o0; output[1] = o1; output[2] = o2; output[3] = o3;
10235 output += 4;
10236 stbIF0( input0 += 4; ) stbIF1( input1 += 4; ) stbIF2( input2 += 4; ) stbIF3( input3 += 4; ) stbIF4( input4 += 4; ) stbIF5( input5 += 4; ) stbIF6( input6 += 4; ) stbIF7( input7 += 4; )
10237 }
10238 #endif
10239 STBIR_NO_UNROLL_LOOP_START
10240 while ( input0 < input0_end )
10241 {
10242 float o0;
10243 STBIR_NO_UNROLL(output);
10244 #ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
10245 stbIF0( o0 = output[0] + input0[0] * c0s; )
10246 #else
10247 stbIF0( o0 = input0[0] * c0s; )
10248 #endif
10249 stbIF1( o0 += input1[0] * c1s; )
10250 stbIF2( o0 += input2[0] * c2s; )
10251 stbIF3( o0 += input3[0] * c3s; )
10252 stbIF4( o0 += input4[0] * c4s; )
10253 stbIF5( o0 += input5[0] * c5s; )
10254 stbIF6( o0 += input6[0] * c6s; )
10255 stbIF7( o0 += input7[0] * c7s; )
10256 output[0] = o0;
10257 ++output;
10258 stbIF0( ++input0; ) stbIF1( ++input1; ) stbIF2( ++input2; ) stbIF3( ++input3; ) stbIF4( ++input4; ) stbIF5( ++input5; ) stbIF6( ++input6; ) stbIF7( ++input7; )
10259 }
10260}
10261
10262#undef stbIF0
10263#undef stbIF1
10264#undef stbIF2
10265#undef stbIF3
10266#undef stbIF4
10267#undef stbIF5
10268#undef stbIF6
10269#undef stbIF7
10270#undef STB_IMAGE_RESIZE_DO_VERTICALS
10271#undef STBIR__vertical_channels
10272#undef STB_IMAGE_RESIZE_DO_HORIZONTALS
10273#undef STBIR_strs_join24
10274#undef STBIR_strs_join14
10275#undef STBIR_chans
10276#ifdef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
10277#undef STB_IMAGE_RESIZE_VERTICAL_CONTINUE
10278#endif
10279
10280#else // !STB_IMAGE_RESIZE_DO_VERTICALS
10281
10282#define STBIR_chans( start, end ) STBIR_strs_join1(start,STBIR__horizontal_channels,end)
10283
10284#ifndef stbir__2_coeff_only
10285#define stbir__2_coeff_only() \
10286 stbir__1_coeff_only(); \
10287 stbir__1_coeff_remnant(1);
10288#endif
10289
10290#ifndef stbir__2_coeff_remnant
10291#define stbir__2_coeff_remnant( ofs ) \
10292 stbir__1_coeff_remnant(ofs); \
10293 stbir__1_coeff_remnant((ofs)+1);
10294#endif
10295
10296#ifndef stbir__3_coeff_only
10297#define stbir__3_coeff_only() \
10298 stbir__2_coeff_only(); \
10299 stbir__1_coeff_remnant(2);
10300#endif
10301
10302#ifndef stbir__3_coeff_remnant
10303#define stbir__3_coeff_remnant( ofs ) \
10304 stbir__2_coeff_remnant(ofs); \
10305 stbir__1_coeff_remnant((ofs)+2);
10306#endif
10307
10308#ifndef stbir__3_coeff_setup
10309#define stbir__3_coeff_setup()
10310#endif
10311
10312#ifndef stbir__4_coeff_start
10313#define stbir__4_coeff_start() \
10314 stbir__2_coeff_only(); \
10315 stbir__2_coeff_remnant(2);
10316#endif
10317
10318#ifndef stbir__4_coeff_continue_from_4
10319#define stbir__4_coeff_continue_from_4( ofs ) \
10320 stbir__2_coeff_remnant(ofs); \
10321 stbir__2_coeff_remnant((ofs)+2);
10322#endif
10323
10324#ifndef stbir__store_output_tiny
10325#define stbir__store_output_tiny stbir__store_output
10326#endif
10327
10328static void STBIR_chans( stbir__horizontal_gather_,_channels_with_1_coeff)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__contributors const * horizontal_contributors, float const * horizontal_coefficients, int coefficient_width )
10329{
10330 float const * output_end = output_buffer + output_sub_size * STBIR__horizontal_channels;
10331 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
10332 STBIR_SIMD_NO_UNROLL_LOOP_START
10333 do {
10334 float const * decode = decode_buffer + horizontal_contributors->n0 * STBIR__horizontal_channels;
10335 float const * hc = horizontal_coefficients;
10336 stbir__1_coeff_only();
10337 stbir__store_output_tiny();
10338 } while ( output < output_end );
10339}
10340
10341static void STBIR_chans( stbir__horizontal_gather_,_channels_with_2_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__contributors const * horizontal_contributors, float const * horizontal_coefficients, int coefficient_width )
10342{
10343 float const * output_end = output_buffer + output_sub_size * STBIR__horizontal_channels;
10344 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
10345 STBIR_SIMD_NO_UNROLL_LOOP_START
10346 do {
10347 float const * decode = decode_buffer + horizontal_contributors->n0 * STBIR__horizontal_channels;
10348 float const * hc = horizontal_coefficients;
10349 stbir__2_coeff_only();
10350 stbir__store_output_tiny();
10351 } while ( output < output_end );
10352}
10353
10354static void STBIR_chans( stbir__horizontal_gather_,_channels_with_3_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__contributors const * horizontal_contributors, float const * horizontal_coefficients, int coefficient_width )
10355{
10356 float const * output_end = output_buffer + output_sub_size * STBIR__horizontal_channels;
10357 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
10358 STBIR_SIMD_NO_UNROLL_LOOP_START
10359 do {
10360 float const * decode = decode_buffer + horizontal_contributors->n0 * STBIR__horizontal_channels;
10361 float const * hc = horizontal_coefficients;
10362 stbir__3_coeff_only();
10363 stbir__store_output_tiny();
10364 } while ( output < output_end );
10365}
10366
10367static void STBIR_chans( stbir__horizontal_gather_,_channels_with_4_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__contributors const * horizontal_contributors, float const * horizontal_coefficients, int coefficient_width )
10368{
10369 float const * output_end = output_buffer + output_sub_size * STBIR__horizontal_channels;
10370 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
10371 STBIR_SIMD_NO_UNROLL_LOOP_START
10372 do {
10373 float const * decode = decode_buffer + horizontal_contributors->n0 * STBIR__horizontal_channels;
10374 float const * hc = horizontal_coefficients;
10375 stbir__4_coeff_start();
10376 stbir__store_output();
10377 } while ( output < output_end );
10378}
10379
10380static void STBIR_chans( stbir__horizontal_gather_,_channels_with_5_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__contributors const * horizontal_contributors, float const * horizontal_coefficients, int coefficient_width )
10381{
10382 float const * output_end = output_buffer + output_sub_size * STBIR__horizontal_channels;
10383 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
10384 STBIR_SIMD_NO_UNROLL_LOOP_START
10385 do {
10386 float const * decode = decode_buffer + horizontal_contributors->n0 * STBIR__horizontal_channels;
10387 float const * hc = horizontal_coefficients;
10388 stbir__4_coeff_start();
10389 stbir__1_coeff_remnant(4);
10390 stbir__store_output();
10391 } while ( output < output_end );
10392}
10393
10394static void STBIR_chans( stbir__horizontal_gather_,_channels_with_6_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__contributors const * horizontal_contributors, float const * horizontal_coefficients, int coefficient_width )
10395{
10396 float const * output_end = output_buffer + output_sub_size * STBIR__horizontal_channels;
10397 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
10398 STBIR_SIMD_NO_UNROLL_LOOP_START
10399 do {
10400 float const * decode = decode_buffer + horizontal_contributors->n0 * STBIR__horizontal_channels;
10401 float const * hc = horizontal_coefficients;
10402 stbir__4_coeff_start();
10403 stbir__2_coeff_remnant(4);
10404 stbir__store_output();
10405 } while ( output < output_end );
10406}
10407
10408static void STBIR_chans( stbir__horizontal_gather_,_channels_with_7_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__contributors const * horizontal_contributors, float const * horizontal_coefficients, int coefficient_width )
10409{
10410 float const * output_end = output_buffer + output_sub_size * STBIR__horizontal_channels;
10411 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
10412 stbir__3_coeff_setup();
10413 STBIR_SIMD_NO_UNROLL_LOOP_START
10414 do {
10415 float const * decode = decode_buffer + horizontal_contributors->n0 * STBIR__horizontal_channels;
10416 float const * hc = horizontal_coefficients;
10417
10418 stbir__4_coeff_start();
10419 stbir__3_coeff_remnant(4);
10420 stbir__store_output();
10421 } while ( output < output_end );
10422}
10423
10424static void STBIR_chans( stbir__horizontal_gather_,_channels_with_8_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__contributors const * horizontal_contributors, float const * horizontal_coefficients, int coefficient_width )
10425{
10426 float const * output_end = output_buffer + output_sub_size * STBIR__horizontal_channels;
10427 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
10428 STBIR_SIMD_NO_UNROLL_LOOP_START
10429 do {
10430 float const * decode = decode_buffer + horizontal_contributors->n0 * STBIR__horizontal_channels;
10431 float const * hc = horizontal_coefficients;
10432 stbir__4_coeff_start();
10433 stbir__4_coeff_continue_from_4(4);
10434 stbir__store_output();
10435 } while ( output < output_end );
10436}
10437
10438static void STBIR_chans( stbir__horizontal_gather_,_channels_with_9_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__contributors const * horizontal_contributors, float const * horizontal_coefficients, int coefficient_width )
10439{
10440 float const * output_end = output_buffer + output_sub_size * STBIR__horizontal_channels;
10441 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
10442 STBIR_SIMD_NO_UNROLL_LOOP_START
10443 do {
10444 float const * decode = decode_buffer + horizontal_contributors->n0 * STBIR__horizontal_channels;
10445 float const * hc = horizontal_coefficients;
10446 stbir__4_coeff_start();
10447 stbir__4_coeff_continue_from_4(4);
10448 stbir__1_coeff_remnant(8);
10449 stbir__store_output();
10450 } while ( output < output_end );
10451}
10452
10453static void STBIR_chans( stbir__horizontal_gather_,_channels_with_10_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__contributors const * horizontal_contributors, float const * horizontal_coefficients, int coefficient_width )
10454{
10455 float const * output_end = output_buffer + output_sub_size * STBIR__horizontal_channels;
10456 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
10457 STBIR_SIMD_NO_UNROLL_LOOP_START
10458 do {
10459 float const * decode = decode_buffer + horizontal_contributors->n0 * STBIR__horizontal_channels;
10460 float const * hc = horizontal_coefficients;
10461 stbir__4_coeff_start();
10462 stbir__4_coeff_continue_from_4(4);
10463 stbir__2_coeff_remnant(8);
10464 stbir__store_output();
10465 } while ( output < output_end );
10466}
10467
10468static void STBIR_chans( stbir__horizontal_gather_,_channels_with_11_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__contributors const * horizontal_contributors, float const * horizontal_coefficients, int coefficient_width )
10469{
10470 float const * output_end = output_buffer + output_sub_size * STBIR__horizontal_channels;
10471 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
10472 stbir__3_coeff_setup();
10473 STBIR_SIMD_NO_UNROLL_LOOP_START
10474 do {
10475 float const * decode = decode_buffer + horizontal_contributors->n0 * STBIR__horizontal_channels;
10476 float const * hc = horizontal_coefficients;
10477 stbir__4_coeff_start();
10478 stbir__4_coeff_continue_from_4(4);
10479 stbir__3_coeff_remnant(8);
10480 stbir__store_output();
10481 } while ( output < output_end );
10482}
10483
10484static void STBIR_chans( stbir__horizontal_gather_,_channels_with_12_coeffs)( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__contributors const * horizontal_contributors, float const * horizontal_coefficients, int coefficient_width )
10485{
10486 float const * output_end = output_buffer + output_sub_size * STBIR__horizontal_channels;
10487 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
10488 STBIR_SIMD_NO_UNROLL_LOOP_START
10489 do {
10490 float const * decode = decode_buffer + horizontal_contributors->n0 * STBIR__horizontal_channels;
10491 float const * hc = horizontal_coefficients;
10492 stbir__4_coeff_start();
10493 stbir__4_coeff_continue_from_4(4);
10494 stbir__4_coeff_continue_from_4(8);
10495 stbir__store_output();
10496 } while ( output < output_end );
10497}
10498
10499static void STBIR_chans( stbir__horizontal_gather_,_channels_with_n_coeffs_mod0 )( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__contributors const * horizontal_contributors, float const * horizontal_coefficients, int coefficient_width )
10500{
10501 float const * output_end = output_buffer + output_sub_size * STBIR__horizontal_channels;
10502 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
10503 STBIR_SIMD_NO_UNROLL_LOOP_START
10504 do {
10505 float const * decode = decode_buffer + horizontal_contributors->n0 * STBIR__horizontal_channels;
10506 int n = ( ( horizontal_contributors->n1 - horizontal_contributors->n0 + 1 ) - 4 + 3 ) >> 2;
10507 float const * hc = horizontal_coefficients;
10508
10509 stbir__4_coeff_start();
10510 STBIR_SIMD_NO_UNROLL_LOOP_START
10511 do {
10512 hc += 4;
10513 decode += STBIR__horizontal_channels * 4;
10514 stbir__4_coeff_continue_from_4( 0 );
10515 --n;
10516 } while ( n > 0 );
10517 stbir__store_output();
10518 } while ( output < output_end );
10519}
10520
10521static void STBIR_chans( stbir__horizontal_gather_,_channels_with_n_coeffs_mod1 )( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__contributors const * horizontal_contributors, float const * horizontal_coefficients, int coefficient_width )
10522{
10523 float const * output_end = output_buffer + output_sub_size * STBIR__horizontal_channels;
10524 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
10525 STBIR_SIMD_NO_UNROLL_LOOP_START
10526 do {
10527 float const * decode = decode_buffer + horizontal_contributors->n0 * STBIR__horizontal_channels;
10528 int n = ( ( horizontal_contributors->n1 - horizontal_contributors->n0 + 1 ) - 5 + 3 ) >> 2;
10529 float const * hc = horizontal_coefficients;
10530
10531 stbir__4_coeff_start();
10532 STBIR_SIMD_NO_UNROLL_LOOP_START
10533 do {
10534 hc += 4;
10535 decode += STBIR__horizontal_channels * 4;
10536 stbir__4_coeff_continue_from_4( 0 );
10537 --n;
10538 } while ( n > 0 );
10539 stbir__1_coeff_remnant( 4 );
10540 stbir__store_output();
10541 } while ( output < output_end );
10542}
10543
10544static void STBIR_chans( stbir__horizontal_gather_,_channels_with_n_coeffs_mod2 )( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__contributors const * horizontal_contributors, float const * horizontal_coefficients, int coefficient_width )
10545{
10546 float const * output_end = output_buffer + output_sub_size * STBIR__horizontal_channels;
10547 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
10548 STBIR_SIMD_NO_UNROLL_LOOP_START
10549 do {
10550 float const * decode = decode_buffer + horizontal_contributors->n0 * STBIR__horizontal_channels;
10551 int n = ( ( horizontal_contributors->n1 - horizontal_contributors->n0 + 1 ) - 6 + 3 ) >> 2;
10552 float const * hc = horizontal_coefficients;
10553
10554 stbir__4_coeff_start();
10555 STBIR_SIMD_NO_UNROLL_LOOP_START
10556 do {
10557 hc += 4;
10558 decode += STBIR__horizontal_channels * 4;
10559 stbir__4_coeff_continue_from_4( 0 );
10560 --n;
10561 } while ( n > 0 );
10562 stbir__2_coeff_remnant( 4 );
10563
10564 stbir__store_output();
10565 } while ( output < output_end );
10566}
10567
10568static void STBIR_chans( stbir__horizontal_gather_,_channels_with_n_coeffs_mod3 )( float * output_buffer, unsigned int output_sub_size, float const * decode_buffer, stbir__contributors const * horizontal_contributors, float const * horizontal_coefficients, int coefficient_width )
10569{
10570 float const * output_end = output_buffer + output_sub_size * STBIR__horizontal_channels;
10571 float STBIR_SIMD_STREAMOUT_PTR( * ) output = output_buffer;
10572 stbir__3_coeff_setup();
10573 STBIR_SIMD_NO_UNROLL_LOOP_START
10574 do {
10575 float const * decode = decode_buffer + horizontal_contributors->n0 * STBIR__horizontal_channels;
10576 int n = ( ( horizontal_contributors->n1 - horizontal_contributors->n0 + 1 ) - 7 + 3 ) >> 2;
10577 float const * hc = horizontal_coefficients;
10578
10579 stbir__4_coeff_start();
10580 STBIR_SIMD_NO_UNROLL_LOOP_START
10581 do {
10582 hc += 4;
10583 decode += STBIR__horizontal_channels * 4;
10584 stbir__4_coeff_continue_from_4( 0 );
10585 --n;
10586 } while ( n > 0 );
10587 stbir__3_coeff_remnant( 4 );
10588
10589 stbir__store_output();
10590 } while ( output < output_end );
10591}
10592
10593static stbir__horizontal_gather_channels_func * STBIR_chans(stbir__horizontal_gather_,_channels_with_n_coeffs_funcs)[4]=
10594{
10595 STBIR_chans(stbir__horizontal_gather_,_channels_with_n_coeffs_mod0),
10596 STBIR_chans(stbir__horizontal_gather_,_channels_with_n_coeffs_mod1),
10597 STBIR_chans(stbir__horizontal_gather_,_channels_with_n_coeffs_mod2),
10598 STBIR_chans(stbir__horizontal_gather_,_channels_with_n_coeffs_mod3),
10599};
10600
10601static stbir__horizontal_gather_channels_func * STBIR_chans(stbir__horizontal_gather_,_channels_funcs)[12]=
10602{
10603 STBIR_chans(stbir__horizontal_gather_,_channels_with_1_coeff),
10604 STBIR_chans(stbir__horizontal_gather_,_channels_with_2_coeffs),
10605 STBIR_chans(stbir__horizontal_gather_,_channels_with_3_coeffs),
10606 STBIR_chans(stbir__horizontal_gather_,_channels_with_4_coeffs),
10607 STBIR_chans(stbir__horizontal_gather_,_channels_with_5_coeffs),
10608 STBIR_chans(stbir__horizontal_gather_,_channels_with_6_coeffs),
10609 STBIR_chans(stbir__horizontal_gather_,_channels_with_7_coeffs),
10610 STBIR_chans(stbir__horizontal_gather_,_channels_with_8_coeffs),
10611 STBIR_chans(stbir__horizontal_gather_,_channels_with_9_coeffs),
10612 STBIR_chans(stbir__horizontal_gather_,_channels_with_10_coeffs),
10613 STBIR_chans(stbir__horizontal_gather_,_channels_with_11_coeffs),
10614 STBIR_chans(stbir__horizontal_gather_,_channels_with_12_coeffs),
10615};
10616
10617#undef STBIR__horizontal_channels
10618#undef STB_IMAGE_RESIZE_DO_HORIZONTALS
10619#undef stbir__1_coeff_only
10620#undef stbir__1_coeff_remnant
10621#undef stbir__2_coeff_only
10622#undef stbir__2_coeff_remnant
10623#undef stbir__3_coeff_only
10624#undef stbir__3_coeff_remnant
10625#undef stbir__3_coeff_setup
10626#undef stbir__4_coeff_start
10627#undef stbir__4_coeff_continue_from_4
10628#undef stbir__store_output
10629#undef stbir__store_output_tiny
10630#undef STBIR_chans
10631
10632#endif // HORIZONALS
10633
10634#undef STBIR_strs_join2
10635#undef STBIR_strs_join1
10636
10637#endif // STB_IMAGE_RESIZE_DO_HORIZONTALS/VERTICALS/CODERS
10638
10639/*
10640------------------------------------------------------------------------------
10641This software is available under 2 licenses -- choose whichever you prefer.
10642------------------------------------------------------------------------------
10643ALTERNATIVE A - MIT License
10644Copyright (c) 2017 Sean Barrett
10645Permission is hereby granted, free of charge, to any person obtaining a copy of
10646this software and associated documentation files (the "Software"), to deal in
10647the Software without restriction, including without limitation the rights to
10648use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
10649of the Software, and to permit persons to whom the Software is furnished to do
10650so, subject to the following conditions:
10651The above copyright notice and this permission notice shall be included in all
10652copies or substantial portions of the Software.
10653THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
10654IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
10655FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
10656AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
10657LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
10658OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
10659SOFTWARE.
10660------------------------------------------------------------------------------
10661ALTERNATIVE B - Public Domain (www.unlicense.org)
10662This is free and unencumbered software released into the public domain.
10663Anyone is free to copy, modify, publish, use, compile, sell, or distribute this
10664software, either in source code form or as a compiled binary, for any purpose,
10665commercial or non-commercial, and by any means.
10666In jurisdictions that recognize copyright laws, the author or authors of this
10667software dedicate any and all copyright interest in the software to the public
10668domain. We make this dedication for the benefit of the public at large and to
10669the detriment of our heirs and successors. We intend this dedication to be an
10670overt act of relinquishment in perpetuity of all present and future rights to
10671this software under copyright law.
10672THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
10673IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
10674FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
10675AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
10676ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
10677WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
10678------------------------------------------------------------------------------
10679*/
