Network interface
The network interface implementation is discussed in this page.
The Network Interface (NI) has the role of abstracting the network communication details, providing a high-level interface to all the components in the tile which require communication. The Router Interface splits application messages (such as coherence related messages) into multiple flits to be injected in the local router.
The virtual channels used by tile components are reported below.
Due to the coherence protocol implemented, both the directory controller and the cache controller might require access to the Response virtual channel. This is carefully handled by the network interface, which stores incoming requests in FIFOs and arbitrating concurrent requests.
Another main feature supported is multicast addressing. The directory controller, in some cases, might send the same packet to multiple recipients (e.g. ACKs). In the current implementation, it is handled by the Network Interface as a sequence of unicast messages.
Contents
General architecture
The network interface has a regular internal structure. This is because ejection and injection services are implemented as two distinct functionalities, allowing a modular design. Moreover, the ejection and injection logics are almost the same for every virtual channel.
For this reason, two separated modules are provided, that together handle the whole network communication for a single virtual channel:
-
virtual_network_core_to_net, which handles flit injection; -
virtual_network_net_to_core, which handles ejection.
Both of them are parameterized and easily adapt to different virtual channel needs. In particular, the virtual channels number and the packet length must be specified.
As the router ejects flits toward the tile, only the network interface assigned to that specific virtual channel reassembles them back and buffers the reconstructed packet until the corresponding component is ready to work. The virtual channel 0 instantiation is reported below.
// --- Request Virtual Network VC0 --- // virtual_network_net_to_core # ( .VCID ( VC0 ), .PACKET_BODY_SIZE ( $bits ( coherence_request_message_t ) ), .FLIT_NUMB ( `RESP_FLIT_NUMB ), ... ... ) request_virtual_network_net_to_core ( ... //Cache Controller interface .vn_ntc_packet_out ( ni_request ), .vn_ntc_packet_valid ( ni_request_valid ), .core_packet_consumed ( dc_request_consumed ), //Router interface .vn_ntc_credit ( ni_credit[VC0] ), .router_flit_valid ( router_flit_in_valid ), .router_flit_in ( router_flit_in ) );
Note that the packet body size parameter is linked with the flit number parameter, but the module handles them separately.
On the other side, the injection logic buffers outgoing packets, splitting them into flits and competes with the others VCs to obtain access to the unique router local port. To grant access to the local injection port, a round-robin arbiter with a grant-and-hold circuitry has been developed. The granted virtual channel index is used as a selector in a multiplexer which sends the right flit to the router.
assign vno_requests =
{vn_packet_pending[ VC3 ] & ~router_credit[VC3],
vn_packet_pending[ VC2 ] & ~router_credit[VC2],
vn_packet_pending[ VC1 ] & ~router_credit[VC1],
vn_packet_pending[ VC0 ] & ~router_credit[VC0]};
rr_arbiter # (
.NUM_REQUESTERS ( `VC_PER_PORT )
)
ni_request_rr_arbiter (
.clk ( clk ),
.reset ( reset ),
.request ( vno_requests ),
.update_lru ( 1'b1 ),
.grant_oh ( vno_granted )
) ;
oh_to_idx # (
.NUM_SIGNALS ( `VC_PER_PORT ),
.DIRECTION ( "LSB0" )
)
ni_request_grant_oh_to_idx (
.one_hot ( vno_granted ),
.index ( vco_granted_id )
);
assign ni_flit_out = vn_flit_out[vco_granted_id],
ni_flit_out_valid = vn_flit_valid[vco_granted_id];
Special care must be given to the response virtual channel. Two injection and two ejection modules are instanced for this virtual channel, each interfacing respectively with the cache controller and the directory controller.
The ejection module supports a specific parameter to let it know if it is interfacing the directory or the cache controller. When this parameter is set, it will check the core_destination field in the flit (see flit structure) to know if it should ignore it.
The injection modules are not aware of who is using them. For this reason, both will compete to be granted access to the virtual channel (and the winner will compete with the others to access router input port). A round-robin arbiter with a grant-and-hold circuitry is used. This will ensure that once one of the controllers gains access to the virtual channel, it will retain it until the full request has been sent.
// each bit is high respectively if the Cache Controller or the Directory Controller wants to inject a packet
assign pending_tmp = {response_in[CC_ID].vn_packet_pending, response_in[DC_ID].vn_packet_pending};
// the arbiter chooses among the two of them
grant_hold_rr_arbiter #(
.NUM_REQUESTERS( 2 )
)
response_vn_rr_arbiter (
.clk ( clk ),
.reset ( reset ),
.request ( pending_tmp ),
.hold_in ( pending_tmp ),
.grant_oh ( response_vn_grant_oh )
);
// the arbitration result is used to select the winning flit, and the signals are updated accordingly
assign vn_packet_pending[ VC1 ] = |response_vn_grant_oh ;
assign vn_flit_out[VC1] = response_vn_grant_oh[0]? response_in[DC_ID].vn_flit_out : response_in[CC_ID].vn_flit_out;
assign vn_flit_valid[VC1] = response_vn_grant_oh[0]? response_in[DC_ID].vn_flit_valid : response_in[CC_ID].vn_flit_valid;
Network to core module
This module is composed of two parts: a control unit which handles the incoming flits, rebuilding them as a packet; and a queue of rebuilt packets.
Control unit
The control unit has a logic based on registers used to store temporary results of the rebuilding process.
Each register tracks the count of flits already received, and uses this count to merge them into a packet.
logic [$clog2( FLIT_NUMB ) - 1 : 0] count;
flit_body_t [FLIT_NUMB - 1 : 0] rebuilt_packet;
...
if (router_flit_valid) begin
rebuilt_packet[count] <= router_flit_in.payload;
if (router_flit_in.flit_type == TAIL || router_flit_in.flit_type == HT) begin
count <= '{default: '0};
cu_packet_rebuilt_compl <= 1'b1;
end else
count <= count + 1;
...
Please note that the control logic also detects if the flits are for the cache controller or to the directory controller, that are connected on the same response virtual channel.
if (router_flit_in.flit_type == HEADER || router_flit_in.flit_type == HT) begin cu_is_for_cc <= router_flit_in.core_destination == TO_CC; cu_is_for_dc <= router_flit_in.core_destination == TO_DC; end
Rebuilt packets queue
Rebuilt packets are stored in a FIFO, so we can enqueue multiple requests. When the receiver component is ready to handle the request, it will assert the core_packet_consumed signal, freeing one buffer slot.
The back-pressure signals will be raised when there are two free buffer slots. This accounts for the worst case, when there is a sequence of 1-flit packets incoming, some of which are yet in the pipe stages and should not be lost. The pipe stages in between are two: the router crossbar and the control unit of this module.
sync_fifo #( .WIDTH ( PACKET_BODY_SIZE ), .SIZE ( PACKET_FIFO_SIZE ), .ALMOST_FULL_THRESHOLD ( PACKET_FIFO_SIZE - 2 ) ) rebuilt_packet_fifo ( ... .almost_full ( packet_alm_fifo_full ), .enqueue_en ( enqueue_en ), .value_i ( cu_rebuilt_packet ), .empty ( rebuilt_packet_fifo_empty ), .almost_empty( ), .dequeue_en ( core_packet_consumed ), .value_o ( vn_ntc_packet_out ) ); assign vn_ntc_credit = packet_alm_fifo_full;
The enqueue signal is generated from the incoming flit virtual channel ID. For the reason explained above, the only exception are that of cache and directory controllers. In that case, the module will also check that the flit is for the cache/directory controller, and it will enqueue it based on the TYPE parameter.
generate if ( TYPE == "CC" ) assign enqueue_en = cu_packet_rebuilt_compl & cu_is_for_cc; else if ( TYPE == "DC" ) assign enqueue_en = cu_packet_rebuilt_compl & cu_is_for_dc; else assign enqueue_en = cu_packet_rebuilt_compl; endgenerate
Example
If the incoming packet arrives as this sequence of flits:
1st Flit = {FLIT_TYPE_HEAD, FLIT_BODY_SIZE'h20}
2nd Flit = {FLIT_TYPE_BODY, FLIT_BODY_SIZE'h40}
3rd Flit = {FLIT_TYPE_BODY, FLIT_BODY_SIZE'h60}
4th Flit = {FLIT_TYPE_TAIL, FLIT_BODY_SIZE'h10};
The rebuilt packet passed to the Cache Controller is:
Packet = {FLIT_BODY_SIZE'h10, FLIT_BODY_SIZE'h60, FLIT_BODY_SIZE'h40, FLIT_BODY_SIZE'h20}
Core to network module
This module splits a packet into flits and sends them to the router local port. It also supports multicasting, implemented as multiple unicast messages. It is composed of two parts: a packet queue, and a control unit which handles the outgoing flits.
Request queue
Incoming requests are enqueued in a FIFO as the control unit handles them. It also provides stop signals for the requesting device, in case the FIFO gets full.
The enqueued structure is composed of a packet body along with all the recipients of the message.
typedef struct packed {
logic [PACKET_BODY_SIZE - 1 : 0] packet_body;
logic packet_has_data;
tile_address_t [DEST_NUMB - 1 : 0 ] packet_destinations;
logic [DEST_NUMB - 1 : 0 ] packet_destinations_valid;
} packet_information_t;
A request will be enqueued when the requester asserts the packet_valid signal, and the head of the FIFO will be dequeued when the control unit notifies completion.
sync_fifo # ( .WIDTH ( $bits ( packet_information_t ) ), .SIZE ( PACKET_FIFO_SIZE ), .ALMOST_FULL_THRESHOLD ( PACKET_ALMOST_FULL_THRESHOLD ) ) packet_in_fifo ( ... .almost_full ( vn_packet_fifo_full ), .enqueue_en ( packet_valid ), .value_i ( packet_information_in ), .empty ( packet_fifo_empty ), .almost_empty ( ), .dequeue_en ( cu_packet_dequeue ), .value_o ( packet_information_out ) ) ;
Request signals are generated, which allow this module to compete for router port access.
assign packet_pending = ~packet_fifo_empty;
Control unit
Control unit's responsibilities are:
- split a packet into flits;
- determine the type (head, body, tail, head-tail) of each flit;
- calculate a pre-routing of the outgoing flits, as routers implement routing look-ahead (see Network);
- properly handle multicast messages, if required;
- account for router back-pressure signals;
control_unit_packet_to_flit # ( parameter DEST_OH = "TRUE", ... parameter PACKET_BODY_SIZE = 256, parameter DEST_NUMB = 4 ) ( ... input logic packet_valid, input logic packet_has_data, input tile_address_t [DEST_NUMB - 1 : 0] packet_destinations, input logic [DEST_NUMB - 1 : 0] packet_destinations_valid, input logic flit_credit, output logic [$clog2 ( PACKET_BODY_SIZE ) - 1 : 0] cu_packet_chunck_sel, output logic cu_flit_valid, output flit_header_t cu_flit_out_header, output logic cu_packet_dequeue );
A request is considered fulfilled when a unicast message has been sent to all its recipients. Recipients are served in a round-robin fashion.
rr_arbiter # ( .NUM_REQUESTERS ( DEST_NUMB ) ) rr_arbiter ( .clk ( clk ) , .reset ( reset ) , .request ( packet_dest_pending ) , .update_lru ( 1'b0 ) , .grant_oh ( destination_grant_oh ) );
Already served recipients are stored in a bit mask, which gets updated with the grant signal after each sending.
dest_served <= dest_served | destination_grant_oh;
This bit mask tracks remaining recipients.
assign packet_dest_pending = packet_destinations_valid & ~dest_served; assign packet_has_dest_pending = |packet_dest_pending;
Routing is done for each recipient, based on the destination address.
The parameter DEST_OH determines how multicast addresses are generated. If DEST_OH is true, the module will consider the input signal packet_destination as a one-hot encoded bit mask, where every tile has a corresponding position in this mask. Otherwise, if DEST_OH is false, multicast addresses are passed to the module into the packet_destinations input signal, and packet_destinations_valid is used to know which position into the array contains a valid address.
generate if ( DEST_OH == "TRUE" ) begin assign real_dest.x = destination_grant_id[`TOT_X_NODE_W - 1 : 0 ], real_dest.y = destination_grant_id[`TOT_Y_NODE_W + `TOT_X_NODE_W - 1 : `TOT_X_NODE_W]; end else begin assign real_dest = packet_destinations[destination_grant_id]; end endgenerate
A finite state machine handles the remaining required actions, keeping count of how many flits have been sent up until now, and choosing the flit payload among the current packet chunk to be sent.
