<p>To map a continuous range (obtained for example as the output of a pseudo-random number generator, or alternatively the logistic map) to a small set of discrete values, you would need to split the continuous range into regions, and assign an output value to each of those regions. The limits of those regions would determine the corresponding threshold values to use.</p>

<p>For example, in the binary case you start off with a continuous range of values in <code>[0,1]</code> which you split into two regions: <code>[0,0.5]</code> and <code>(0.5,1]</code>. Each of those region begin assigned an output symbol, namely <code>-1</code> and <code>+1</code>. As you have noted, the boundary of the regions being set to the midpoint of your <code>[0,1]</code> input range gives you a threshold of 0.5. This could be implemented as:</p>

<pre><code>if (x &gt; 0.5)
  symbol = +1;
else
  symbol = -1;
end
</code></pre>

<p>As a more compact implementation, the formula <code>2*(x&gt;0.5)-1</code> takes advantage of the fact that in Matlab a <em>true</em> condition (from the <code>x&gt;0.5</code> expression) has a value of 1, whereas <em>false</em> has a value of 0.</p>

<p>For 3 discrete output values, you'd similarly split your <code>[0,1]</code> input range into 3 regions: <code>[0,1/3]</code>, <code>(1/3,2/3]</code> and <code>(2/3,1]</code>. The corresponding thresholds thus being 1/3 and 2/3.</p>

<p>Finally for 8 discrete output values, you would similarly split your <code>[0,1]</code> input range into 8 regions: <code>[0,1/8]</code>, <code>(1/8,2/8]</code>, <code>(2/8,3/8]</code>, <code>(3/8,4/8]</code>, <code>(4/8,5/8]</code>, <code>(5/8,6/8]</code>, <code>(6/8,7/8]</code> and <code>(7/8,1]</code>. The corresponding thresholds thus being 1/8, 2/8, 3/8, 4/8, 5/8, 6/8 and 7/8, as illustrated in the following diagram:</p>

<pre><code>thresholding function input: |-----|-----|-----|-----|-----|-----|-----|-----|
                             0     |     |     |     |     |     |     |     1
thresholds:                       1/8   2/8   3/8   4/8   5/8   6/8   7/8
                                |     |     |     |     |     |     |     |
                                v     v     v     v     v     v     v     v
generated symbol:              -7    -5    -3    -1    +1    +3    +5    +7
</code></pre>

<p>This then gives the following symbol mapping implementation:</p>

<pre><code>if (x &lt; 1/8)
  symbol = -7;
elseif (x &lt; 2/8)
  symbol = -5;
elseif (x &lt; 3/8)
  symbol = -3;
elseif (x &lt; 4/8)
  symbol = -1;
elseif (x &lt; 5/8)
  symbol = +1;
elseif (x &lt; 6/8)
  symbol = +3;
elseif (x &lt; 7/8)
  symbol = +5;
else
  symbol = +7;
end
</code></pre>

<p>As a more compact implementation, you could similarly use the <code>floor</code> function to obtain discrete levels:</p>

<pre><code>% x : some value in the [0,1] range
% s : a symbol in the {-7,-5,-3,-1,+1,+3,+5,+7} set
function s = threshold(x)
  % Note on implementation:
  %   8*x turns the input range from [0,1] to [0,8]
  %   floor(8*x) then turns that into values {0,1,2,3,4,5,6,7}
  %   then a linear transform (2*() - 7) is applied to map
  %      0 -&gt; -7, 1 -&gt; -5, 2 -&gt; -3, ..., 7 -&gt; 7
  %   min/max finally applied just as a safety to make sure we don't overflow due
  %   to roundoff errors (if any).
  s = min(7, max(-7, 2*floor(8*x) - 7));
end
</code></pre>

<p>Now if you want to generate complex symbols with 8 levels for the real part and 8 levels for the imaginary part, you'd simply combine them just like in the binary case. Mainly you'd generate a first value which gives you the real part, then a second value for the imaginary part:</p>

<pre><code>x_real = rand(); % random input 0 &lt;= x_real &lt;= 1
x_imag = rand(); % another one
s = threshold(x_real) + sqrt(-1)*threshold(x_imag);
</code></pre>

<hr>

<p><strong>Addressing some points raised by a previous revision of the question:</strong></p>

<p>One thing to note is that <code>x[n+1] = 4*x[n](1-x[n])</code> maps values in <code>[0,1]</code> to the same range of values. This makes it possible to iteratively apply the mapping to obtain additional values, and correspondingly generate a binary sequence with the threshold application (<code>x &gt; 0.5</code>). The function <code>f(x)</code> you provided (in an earlier edit of the question) on the other hand, maps values within a range with discontinuities (roughly covering <code>[-7.5,7.5]</code> depending on <code>p</code>) to <code>[0,1]</code>. In other words you would need to either modify <code>f(x)</code> or otherwise map its output back to the input domain of <code>f(x)</code>. It would probably be easier to consider a general uniform pseudo-random number generator over the <code>[-8,+8]</code> range as input to the threshold function:</p>

<pre><code>% x : some value in the [-8,8] range
% s : a symbol in the {-7,-5,-3,-1,+1,+3,+5,+7} set
function s = threshold_8PAM(x)
  s = min(7, max(-7, 2*round(x/2 + 0.5) - 1));
end
</code></pre>

<p>To get the final 64-QAM symbols you would combine two 8-PAM symbols in quadrature (i.e. <code>x64qam = xQ + sqrt(-1)*xI</code>, where <code>xQ</code> and <code>xI</code> have both been generated with the above procedure).</p>

<p>That said, if the goal is to implement a digital communication system using 64-QAM symbols with additional chaotic modulation, you'd ultimately want to take into account the source of input data to transmit rather than randomly generating both the chaotic modulation and the source data in one shot. That is even if for performance evaluation you wind up generating the source data randomly, it is still a good idea to be generating it independently of the chaotic modulation.</p>

<p>Addressing those concerns, the paper <a href="http://arxiv.org/abs/1302.3800" rel="nofollow noreferrer"><em>An Enhanced Spectral Efficiency Chaos-Based Symbolic Dynamics Transceiver Design</em></a> suggests a different approach based on the inverse map you provided, which can be implemented as:</p>

<pre><code>function x = inverse_mapping(x,SymbIndex,p)
  if (SymbIndex==0)
    x = ((1-p)*x-14)/2;
  elseif (SymbIndex==1)
    x = ((1-p)*x-10)/2;
  elseif (SymbIndex==2)
    x = ((1-p)*x-6)/2;
  elseif (SymbIndex==3)
    x = ((1-p)*x-2)/2;
  elseif (SymbIndex==4)
    x = ((1-p)*x+2)/2;
  elseif (SymbIndex==5)
    x = ((1-p)*x+6)/2;
  elseif (SymbIndex==6)
    x = ((1-p)*x+10)/2;
  elseif (SymbIndex==7)
    x = ((1-p)*x+14)/2;
  end
end
</code></pre>

<p>As you may notice, the function takes a symbol index (3 bits, which you'd get from the input source data) and the current state of the modulated output (which you may seed with any value within the convergence range of <code>inverse_mapping</code>) as two independent input streams. Note that you can compute the bounds of the convergence range of <code>inverse_mapping</code> by finding the limits of repeated application of the mapping using input symbol index <code>s=0</code>, and <code>s=7</code> (using for example a seed of <code>x=0</code>). This should converge to <code>[-14/(1+p), 14/(1+p)]</code>.</p>

<p>The chaotic modulation described in <a href="http://arxiv.org/abs/1302.3800" rel="nofollow noreferrer">the above referenced paper</a> can then be achieved with (setting the control parameter <code>p=0.8</code> as an example):</p>

<pre><code>% Simulation parameters
Nsymb = 10000;
p     = 0.8;
M     = 64;

% Source data generation
SymbolIndexQ = randi([0 sqrt(M)-1],Nsymb,1);
SymbolIndexI = randi([0 sqrt(M)-1],Nsymb,1);

% Modulation
xmax = 14/(1+p); % found by iterative application of inverse_mapping 
xQ = xmax*(2*rand(1)-1); % seed initial state
xI = xmax*(2*rand(1)-1); % seed initial state
x  = zeros(Nsymb,1);
for i=1:Nsymb
  xQ = inverse_mapping(xQ, SymbolIndexQ(i), p);
  xI = inverse_mapping(xI, SymbolIndexI(i), p);
  x(i) = xQ + sqrt(-1)*xI;
end

% x holds the modulated symbols
plot(real(x), imag(x), '.');

% if you also need the unmodulated symbols you can get them from
% SymbolIndexQ and SymbolIndexI
s = (2*SymbolIndexQ-7) + sqrt(-1)*(2*SymbolIndexI-7);
</code></pre>

<p>with should produce the corresponding constellation diagram:</p>

<p><a href="https://i.stack.imgur.com/TQ0Sf.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/TQ0Sf.png" alt="64QAM with chaotic modulation, p=0.8"></a></p>

<p>or with <code>p=1</code> (which is essentially unmodulated):</p>

<p><a href="https://i.stack.imgur.com/OFmId.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/OFmId.png" alt="64QAM p=1.0"></a></p>
Matlab : Help in implementing an encoding for realizing a mapping function
<p>An example : Consider the unimodal <a href="https://en.wikipedia.org/wiki/Logistic_map" rel="nofollow">logistic map</a> : <code>x[n+1] = 4*x[n](1-x[n])</code>. The map can be used to generate +1/-1 symbols using the technique </p>

<p>I want to extend the above concept using the map <code>f(x)</code> for 3 levels, each level corresponds to a symbol but I am unsure how I can do that.</p>